I would like to see a progress bar on Jupyter notebook while I'm running a compute task using Dask, I'm counting all values of id
column from a large csv file +4GB, so any ideas?
import dask.dataframe as dd
df = dd.read_csv('data/train.csv')
df.id.count().compute()
If you're using the single machine scheduler then do this:
from dask.diagnostics import ProgressBar
ProgressBar().register()
http://dask.pydata.org/en/latest/diagnostics-local.html
If you're using the distributed scheduler then do this:
from dask.distributed import progress
result = df.id.count.persist()
progress(result)
Or just use the dashboard
http://dask.pydata.org/en/latest/diagnostics-distributed.html