[SOLVED] RecordIO: "The header of the MXNet RecordIO record...does not start with a valid magic number"

RecordIO: "The header of the MXNet RecordIO record...does not start with a valid magic number"

Using Linear Learner in Sagemaker with MXNet RecordIO, I get "The header of the MXNet RecordIO record at position 5,089,840 in the dataset does not start with a valid magic number" after fit() has been running 38 minutes.

The file was generated using this code. Note that I tried two ways to upload to S3. I also tried direct upload of the BytesIO as well as upload of a file, shown here.

train_file = 'linear_train.data'

f = io.BytesIO()
smac.write_numpy_to_dense_tensor(f, train_X.astype('float32'), train_y.astype('float32'))
f.seek(0)
 
# Write the stuff
with open(train_file, "wb") as fl:
    fl.write(f.getvalue())

# Alternative for upload
# boto3.Session().resource('s3').Bucket(bucket).Object(os.path.join(prefix, 'train', train_file)).upload_fileobj(f)
boto3.client('s3').upload_file(train_file,
                                  Bucket=bucket,
                                  Key=os.path.join(prefix, 'train', train_file),
                                  ExtraArgs={'ACL': 'bucket-owner-full-control'})

To check that the file was non-corrupt, I downloaded it from S3 and simply read it as follows.

   record = MXRecordIO(fl, 'r')
   while True:
      item = record.read()
      # Here we print the item, break when we reach the end, etc. This confirms that the RecordIO is valid.

So, the file seems OK.

How can I run Linear Learner?

Here is the error message:

Failure reason if any: ClientError: Unable to read data channel 'train'. Requested content-type is 'application/x-recordio-protobuf'. Please verify the data matches the requested content-type. (caused by MXNetError)

Caused by: [17:04:49] /opt/brazil-pkg-cache/packages/AIAlgorithmsCppLibs/AIAlgorithmsCppLibs-2.0.3446.0/AL2012/generic-flavor/src/src/aialgs/io/iterator_base.cpp:100: (Input Error) The header of the MXNet RecordIO record at position 5,089,840 in the dataset does not start with a valid magic number.

Stack trace returned 10 entries:
[bt] (0) /opt/amazon/lib/libaialgs.so(+0xbca0) [0x7f337885cca0]
[bt] (1) /opt/amazon/lib/libaialgs.so(+0xbffa) [0x7f337885cffa]
[bt] (2) /opt/amazon/lib/libaialgs.so(aialgs::iterator_base::Next()+0x4a6) [0x7f33788675e6]
[bt] (3) /opt/amazon/lib/libmxnet.so(MXDataIterNext+0x21) [0x7f3367272141]
[bt] (4) /opt/amazon/python2.7/lib/python2.7/lib-dynload/_ctypes.so(ffi_call_unix64+0x4c) [0x7f3378893958]
[bt] (5) /opt/amazon/python2.7/lib/python2.7/lib-dynload/_ctypes.so(ffi_call+0x15f) [0x7f33
Failed 39.0 min; Failure reason if any: ClientError: Unable to read data channel 'train'. Requested content-type is 'application/x-recordio-protobuf'. Please verify the data matches the requested content-type. (caused by MXNetError)

Caused by: [17:04:49] /opt/brazil-pkg-cache/packages/AIAlgorithmsCppLibs/AIAlgorithmsCppLibs-2.0.3446.0/AL2012/generic-flavor/src/src/aialgs/io/iterator_base.cpp:100: (Input Error) The header of the MXNet RecordIO record at position 5,089,840 in the dataset does not start with a valid magic number.

Stack trace returned 10 entries:
[bt] (0) /opt/amazon/lib/libaialgs.so(+0xbca0) [0x7f337885cca0]
[bt] (1) /opt/amazon/lib/libaialgs.so(+0xbffa) [0x7f337885cffa]
[bt] (2) /opt/amazon/lib/libaialgs.so(aialgs::iterator_base::Next()+0x4a6) [0x7f33788675e6]
[bt] (3) /opt/amazon/lib/libmxnet.so(MXDataIterNext+0x21) [0x7f3367272141]
[bt] (4) /opt/amazon/python2.7/lib/python2.7/lib-dynload/_ctypes.so(ffi_call_unix64+0x4c) [0x7f3378893958]
[bt] (5) /opt/amazon/python2.7/lib/python2.7/lib-dynload/_ctypes.so(ffi_call+0x15f) [0x7f33

Solution

Is was because of CSV files in the same S3 folder as the RecordIO.