pythonpandasdataframetqdmcudf

how to use tqdm progress bar in dask_cudf and cudf

I can use tqdm progress bar in pandas for example:

tqdm.pandas()
df = df['var'].progress_apply(lambda x: something(x))

can i do same in thing cudf or dask_cudf if not then how can i use tqdm progress bar in it,

Solution

  • Until progress_apply is available, you would have to implement an equivalent yourself (e.g. using apply_chunks). Just a sketch of the code:

    full_size = 100
t = tqdm(total=full_size)
def chunks_generator():
    chunk_size = 5
    for s in range(0,full_size,chunk_size):
        yield s
        t.update(s)

df.apply_chunks(..., chunks=chunks_generator())