[SOLVED] Timeout when running iPython nbconvert from command line

Timeout when running iPython nbconvert from command line

I'm trying to automate IPython scripts using batch files and the Windows Task Manager. After much digging I've found that you can use Task Manager to trigger a BAT and through that execute the nbconvert package in order to run the Jupyter Notebook without the interactive bits.

In the BAT my code first "turns the Windows cmd into an Anaconda Prompt" with the code: <anaconda3 path>\Scripts\activate.bat <anaconda3 path>

(No formal CS education, so bear with me on this.)

That line allows me to run conda and jupyter commands out of the Windows cmd. I then run

jupyter nbconvert --execute --ExecutePreprocessor.enabled=True --ExecutePreprocessor.timeout=60 <Notebook>

I keep receiving a timeout error when this occurs. Yes, I could adjust the timer, but before setting the timer to 10 hours and coming back in the morning I'm hoping for some understanding of what is going on behind the scenes in nbconvert, and maybe some insights into how to get this working. The .ipynb file works fine when run through Jupyter Notebook. There is however a point at which the script is attempting to load several gb of data, which has taken ~20 minutes each time I've run it.

To that end - when nbconvert engages the Pre-processors, are they running the entire script preemptively? I'm struggling to find a reason as to why nothing opens, pings, runs, updates, etc... when executing the nbconvert command. I would expect at least an uptick in RAM usage, but as far as I can tell that isn't happening either.

How can I work around this, and what is going on behind the scenes when nbconvert is executed?

Traceback:

C:\Users\User>jupyter nbconvert --execute "C:\Users\User\Desktop\_for_Python\CBSA.ipynb"  
[NbConvertApp] Converting notebook C:\Users\User\Desktop\_for_Python\CBSA.ipynb to html  
[NbConvertApp] Executing notebook with kernel: python3  
[NbConvertApp] ERROR | Timeout waiting for execute reply (30s).  
Traceback (most recent call last):  
  File "C:\Users\User\AppData\Local\Continuum\anaconda3\lib\site-packages\nbconvert\preprocessors\execute.py", line 423, in _wait_for_reply  
    msg = self.kc.shell_channel.get_msg(timeout=timeout)  
  File "C:\Users\User\AppData\Local\Continuum\anaconda3\lib\site-packages\jupyter_client\blocking\channels.py", line 57, in get_msg  
    raise Empty  
_queue.Empty

Solution

Fixed by adding the Anaconda3/Library/Bin path to c/windows/system32 PATH. Timeout was occurring due to DLL error, being returned as Timeout in command line.