I wrote a Python script that will run indefinitely. It monitors a directory using PyInotify and uses the Multiprocessing module to run any new files created in those directories through an external script. That all works great.
The problem I am having is writing the output to a file. The filename I chose uses the current date (using datetime.now) and should, theoretically, roll on the hour, every hour.
now = datetime.now()
filename = "/data/db/meta/%s-%s-%s-%s.gz" % (now.year, now.month, now.day, now.hour)
with gzip.open(filename, 'ab') as f:
f.write(json.dumps(data) + "\n")
f.close() #Unsure if I need this, here for debug
Unfortunately, when the hour rolls on -- the output stops and never returns. No exceptions are thrown, it just stops working.
total 2.4M
drwxrwxr-x 2 root root 4.0K Sep 8 08:01 .
drwxrwxr-x 4 root root 12K Aug 29 16:04 ..
-rw-r--r-- 1 root root 446K Aug 29 16:59 2016-8-29-16.gz
-rw-r--r-- 1 root root 533K Aug 30 08:59 2016-8-30-8.gz
-rw-r--r-- 1 root root 38K Sep 7 10:59 2016-9-7-10.gz
-rw-r--r-- 1 root root 95K Sep 7 14:59 2016-9-7-14.gz
-rw-r--r-- 1 root root 292K Sep 7 15:59 2016-9-7-15.gz #Manually run
-rw-r--r-- 1 root root 834K Sep 8 08:59 2016-9-8-8.gz
Those files aren't really owned by root, just changed them for public consumption
As you can see, all of the files timestamps end at :59 and the next hour never happens.
Is there something that I should take into consideration when doing this? Is there something that I am missing running a Python script indefinitely?
After taking a peek. It seems as if PyInotify was my problem. See here (https://unix.stackexchange.com/questions/164794/why-doesnt-inotifywatch-detect-changes-on-added-files)
I adjusted your code to change the file name each minute, which speeds up debugging quite a bit and yet still tests the hypothesis.
import datetime
import gzip, time
from os.path import expanduser
while True:
now = datetime.datetime.now()
filename = expanduser("~")+"/%s-%s-%s-%s-%s.gz" % (now.year, now.month, now.day, now.hour, now.minute)
with gzip.open(filename, 'a') as f:
f.write(str(now) + "\n")
f.write("Data Dump here" + "\n")
time.sleep(10)
This seems to run without an issue. Changing the time-zone of my pc was also picked up and dealt with. I would suspect, given the above, that your error may lie elsewhere and some judicious debug printing of values at key points is needed. Try using a more granular file name as above to speed up the debugging.