[SOLVED] bit64 integers with fst

bit64 integers with fst

I have data in a csv containing long integers. I am exchanging this data between csvs and fst files.

For example,

library(bit64)
library(data.table)
library(fst)
library(magrittr)

# Prepare example csvs
DT64_orig <- data.table(x = (c(2345612345679, 1234567890, 8714567890)))
fwrite(DT64_orig, "DT64_orig.csv")

# Read and move to fst
DT64 <- fread("DT64_orig.csv")
write.fst(DT64, "DT64_fst.fst")

DT_fst2 <- 
  read.fst("DT64_fst.fst") %>%
  setDT

# bit64 integers not preserved:
identical(DT_fst2, DT64)

Is there a way to use fst files for data.tables containing bit64 integers

Solution

It looks like fst might be dropping column attributes either when saving or loading (please ask as an issue on fst package). You can put the column types back yourself in the meantime. bit64::integer64 is a plain double under the hood so no bits have been lost. Just the type information telling R how to print the column.

> DT_fst2
               x
1: 1.158886e-311
2: 6.099576e-315
3: 4.305569e-314
> setattr(DT_fst2$x, "class", "integer64")
> DT_fst2
               x
1: 2345612345679
2:    1234567890
3:    8714567890
> identical(DT_fst2, DT64)
[1] TRUE