In a bash script i pipe the output of a command into the input of another using:
$ output_com | input_com
output_com
is slower than input_com
. In my script, before input_com
is ready to operate, some other commands needs to be executed.
In order to speed up things, i would like to start output_com
at the beginning of the script, outputing into some kind of high capacity fifo, do others commands thats needs to be done then execute input_com
with the fifo in input. Something like this:
#!/bin/bash
#
mkfifo /tmp/fifo
output_com > /tmp/fifo &
somecommand1
somecommand2
input_com < /tmp/fifo
Unfortunatelly the buffer size of fifos is too small for my use case so output_com
is quickly blocked before input_com
starts to read.
The volume of data shared between theses to commands can vary from 500MB to 1GB approx.
Is there some mechanism, tool or concept i'm not aware of that could allow output_com
to start to buffer many MBs on the fifo before input_com
starts to read ?
For example, it would be great if the fifo buffer could be baked by a tmp file (hdd used as buffer) but i have not found how to do that. Using a simple file is non blocking and makes input_com
to terminate before output_com
finishes.
There is a tool called buffer that works like a bigger pipe. It isn't installed by default in most distros, but it can be installed from the package manager.
#!/bin/bash
#
mkfifo /tmp/fifo
output_com | buffer -m 1024m -s 512k > /tmp/fifo &
somecommand1
somecommand2
input_com < /tmp/fifo
wait
Sadly the manual does nos specify the maximum number of blocks that buffer
can work with, nor the maximum size of the buffers, but that should give you a 1GiB buffer.