I am developing an application, where I have large data that's continuously been write into ram, and I am trying to quickly read the data out from ram, and write it into NVMe SSD, after write is completed, I re queue ram space to allow it being written.
My current issue is that the data write continuously in an indefinitely time, so eventually my RAM is not large enough to host all the data, it has to be readout and store into disk. Due to the size of data, the write speed criteria is high ( 1.5G/s).
I often see mmap be mentioned to improve I/O efficiency, but mostly read efficiency because it prevents copy large data to DRAM. So I wonder in my use case, is it beneficial to use mmap to map my SSD directly as virtual memory and directly write to it be faster than standard fwrite ?
The problem with mmap for write is that you do not know when the write is completed or even started, this may increase the number of writes the drive has to do to an LBA since the memory got written into but not in a complete 4KB chunk, the write was issued to the disk and then the data was written again to memory and now the page needs to be written again.
If you want it simple your best bet is to use an O_DIRECT file and use the write syscall or aio. If you want the best speed you can use SPDK to get raw access to the NVMe device without the kernel interfering and having a fully zero-copy write.
SPDK is a driver for NVMe devices completely in user-space. This means that you use a kernel driver to map the PCIe BAR to user-space, tell SPDK to attach to the NVMe device and now you can issue raw NVMe commands to the device without any copy. The big advantage over just a plain mmap is that you have full control on what IOs are done, in what order and how many commands are in-flight at a time and their sizes. It means more work for the application but it really gives you the ultimate control and the best performance.