rdata.tablefread

'Embedded nul in string' error when importing csv with fread

I have a large file (3.5G) that I'm trying to import using data.table::fread.

It was originally created from an rpt file that was opened as text and saved as a CSV.

This has worked fine with smaller files (of the same type of data-same columns and all. This one is just for a longer timeframe and wider reach).

When I try and run 

mydata <- fread("mycsv.csv")

I get the error:

Error in fread("mycsv.csv") : embedded nul in string: 'y\0e\0a\0r\0'

What does this mean?

Solution

  • We can remove the null terminators on the command line using something like:

    sed 's/\\0//g' mycsv.csv > mycsv.csv

    Or as suggested by @marbel, fread allows you to pass the sed call inside the text. Such as:

    fread("sed 's/\\0//g' mycsv.csv")