I have two separate Java processes, a producer and a consumer, that behave as follows:
Producer:
write data file
close data file
write hash file
close hash file
Consumer:
wait until hash file exists
read data file
act on data file
close data file
delete data file
delete hash file
The hash file is only used by this code to signal that the data file is complete. The value of the hash is not validated here. Under normal load there should be one handoff every 3 seconds. Under very heavy load, the rate might increase to one handoff every 100 ms. The filesystem is ext3 with data=ordered.
How can I make sure that the consumer never sees an incomplete file?
One option is to verify the contents of the data file against the hash but that seems unnecessarily expensive. I considered using java.nio.channels.FileLock but I am concerned that it won't remove the need for the hash check because if the producer crashes, the OS frees all its locks even though the file may still be incomplete. This is actually a very likely case for producing an incomplete file. Maybe a lock plus a check that the consumer is still running?
The producer can close the file before starting a new file.
The consumer can wait for the next file before starting the previous one.