linuxbashpipefifo

High capacity buffer fifo or use of a file as buffer for a fifo


In a bash script i pipe the output of a command into the input of another using:

$ output_com | input_com

output_com is slower than input_com. In my script, before input_com is ready to operate, some other commands needs to be executed.

In order to speed up things, i would like to start output_com at the beginning of the script, outputing into some kind of high capacity fifo, do others commands thats needs to be done then execute input_com with the fifo in input. Something like this:

#!/bin/bash
#
mkfifo /tmp/fifo
output_com > /tmp/fifo &

somecommand1
somecommand2
input_com < /tmp/fifo

Unfortunatelly the buffer size of fifos is too small for my use case so output_com is quickly blocked before input_com starts to read. The volume of data shared between theses to commands can vary from 500MB to 1GB approx.

Is there some mechanism, tool or concept i'm not aware of that could allow output_com to start to buffer many MBs on the fifo before input_com starts to read ?

For example, it would be great if the fifo buffer could be baked by a tmp file (hdd used as buffer) but i have not found how to do that. Using a simple file is non blocking and makes input_com to terminate before output_com finishes.


Solution

  • There is a tool called buffer that works like a bigger pipe. It isn't installed by default in most distros, but it can be installed from the package manager.

    #!/bin/bash
    #
    mkfifo /tmp/fifo
    output_com | buffer -m 1024m -s 512k > /tmp/fifo &
    
    somecommand1
    somecommand2
    input_com < /tmp/fifo
    wait
    

    Sadly the manual does nos specify the maximum number of blocks that buffer can work with, nor the maximum size of the buffers, but that should give you a 1GiB buffer.