How does IO buffering work in Ruby? How often is data flushed to the underlying stream when using the IO and File classes? How does this compare to OS buffering? What needs to be done to guarantee that given data has been written to disk, before confidently reading it back for processing?
The Ruby IO documentation is not 100% clear on how this buffering works, but this is what you can extract from the documentation:
The relevant methods to look at:
IO.flush: Flushes IO. I also looked at the Ruby source and a call to IO.flush also calls the underlying OS fflush(). This should be enough to get the file cached, but does not guarantee physical data to disk.IO.sync=: If set to true, no Ruby internal buffering is done. Everything is immidiately sent to the OS, and fflush() is called for each write.IO.sync: Returns the current sync setting (true or false).IO.fsync: Flushes both the Ruby buffers + calls fsync() on the OS (if it supports it). This will guarantee a full flush all the way to the physical disk file.IO.close: Closes the Ruby IO and writes pending data to the OS. Note that this does not imply fsync(). The POSIX documentation on close() says that it does NOT guarantee data is physically written to the file. So you need to use an explicit fsync() call for that.Conclusion: flush and/or close should be enough to get the file cached so that it can be read fully by another process or operation. To get the file all the way to the physical media with certainty, you need to call IO.fsync.
Other related methods:
IO.syswrite: Bypass Ruby internal buffers and do a straight OS write. If you use this then do not mix it with IO.read/write.IO.sysread: Same as above, but for reading.