pythonloggingtqdmpython-logging

Making tqdm write to log files


tqdm is a nice python library to keep track of progress through an iterable.

It's default mode of operation is to repeatedly clear a line and redraw with a carriage but this produced quite nasty output when combined with logging. Is there a way I can get this to write to log files periodically rather than using this print?

This is the best I've got is my own hacky implementation:

def my_tqdm(iterable):
    "Like tqdm but us logging. Include estimated time and time taken."
    start = time.time()
    for i, item in enumerate(iterable):
        elapsed = time.time() - start
        rate = elapsed / (i + 1)
        estimated = rate * len(iterable) - elapsed
        num_items = len(iterable)
        LOGGER.info(
            "Processed %d of %d items (%.1f%%) in %.1fs (%.1fs remaining, %.1f s/item)",
            i,
            num_items,
            i / num_items * 100,
            elapsed,
            estimated,
            rate,
        )
        yield item

But it'd be better if I could do this with tqdm itself so that people don't moan at me in code reviews.


Solution

  • You could redirect the outputs of the TQDM progress bar to a null device (e.g. /dev/null), and manually print/log the status bar whenever you want - either on every iteration, or at a certain interval.

    For example:

    import os
    import time
    import logging
    from tqdm import tqdm
    
    LOG_INTERVAL = 5
    
    logging.basicConfig(level=logging.INFO)
    logger = logging.getLogger('tqdm_logger')
    progress_bar = tqdm(range(20), file=open(os.devnull, 'w'))
    
    for i in progress_bar:
        # do something meaningful instead...
        time.sleep(0.1)
    
        if progress_bar.n % LOG_INTERVAL == 0:
            logger.info(str(progress_bar))
    

    This code block will produce the following outputs:

    INFO:tqdm_logger:  0%|          | 0/20 [00:00<?, ?it/s]
    INFO:tqdm_logger: 25%|██▌       | 5/20 [00:00<00:01,  9.59it/s]
    INFO:tqdm_logger: 50%|█████     | 10/20 [00:01<00:01,  9.54it/s]
    INFO:tqdm_logger: 75%|███████▌  | 15/20 [00:01<00:00,  9.54it/s]