I'm working on Twitter Streaming api to get live tweets data.
I can print that data to console. But what I want is to save the data to a file and that data shouldn't be older than 5 minutes.
How do I continuously roll the file that holds the data from last 5 minutes as we can do for log files.
At the same time the file should be accessible for reading.
Is there any way to do that in Python?
I haven't come across such thing where we can mention such duration for which the file can hold specific data.
A search for "python logrotate" finds this: https://docs.python.org/3.7/library/logging.handlers.html#logging.handlers.TimedRotatingFileHandler. It is probably worth giving a try.
If you'd rather code something yourself, managing the tweets in memory (with a collections.deque for example so it's easy to pop old ones and append new ones), then flush this to file once in a while..., (or use sockets, pickle, simply the variable name to pass this data to your analysis function, as already mentioned in other answers).