jupyter-notebookipythonwindows-subsystem-for-linuxmeep

Save the output of a jupyter notebook code as a "xyz.out" file (using MEEP)


I'm following this tutorial for a EM field solver ( https://mpb.readthedocs.io/en/latest/Python_Data_Analysis_Tutorial/ ) I'm running it on python, using jupyter notebook since its easier to debug and convenient. Now is there any way to run the following command unix% python mpb_tri_rods.py >& tri-rods.out While I'm in jupyter notebook? save the results as an .out file and further work on them? Note: I'm also using wsl for this


Solution

  • The most direct way to transpose that command to IPython and Jupyter use would be execute in a notebook the following:

    !python mpb_tri_rods.py >& tri-rods.out
    

    That exclamation point signals to IPython/Jupyter to take everything after the command and run it off in a temporary shell instance.

    I also should add that you want to put the script you call with !python along with the notebook, in the same directory. You could also use a path, but placing it alongside is the easiest.

    This isn't the way though to normally run a Python script inside a running Jupyter Notebook though. There is a more IPython/Jupyter-centric, feature-rich way that is often used, %run, that I will cover below.

    I can illustrate in part what that is though easily. Instead of the command above, try running !python mpb_quad_rods.py >& tri-rods.out. (I'm expecting there is no script called mpb_quad_rods.py.) Jupyter seems to indicate that ran and completed with no special indication anything unusual happened, similar to what you'd see if you ran 2 + 2 in a cell. You get a number next to the cell indicating completion of that cell as the nth cell in this run of the notebook, where n would be whatever number in the sequence you ran this. In the notebook, you don't see any error like you'd see if you ran pint("test") in a cell directly. But if you type cat tri-rods.out in a new cell in your notebook, you should see something like python: can't open file '/home/jovyan/mpb_quad_rods.py': [Errno 2] No such file or directory after you ran !python mpb_quad_rods.py >& tri-rods.out. But having to do that extra cat step here is not very interactive and IPython/Jupyter are designed to be interactive.

    So normally to run a script in in IPython/Jupyter, you'd take advantage of the %run magic command, and do something like %run my_script.py and if there was output from that or an error, it would be shown right in the notebook. The %run magic has additional options and flags you can use that the documentation for it I referenced discusses. But in Jupyter it is more feature rich than that even covers. For example, stderr channel gets represented nicer in pink and you can even run entire separate notebooks inside the current notebook with %run notebook.ipynb.
    (Aside: One of the nicest flags to use with %run is the interactive flag, -i. Imagine in your notebook you had code in a cell to set a variable a to be 5 and you'd already run that in the current notebook session. If you had a script that said print(a) and ran that with %run myscript.py you'd get a name error because by default it runs the script off. But if you ran that with %run -i myscript.py, it would work because it runs it interactive as if running in the current notebook namespace.)

    In your case, the %run command would be something like, %run mpb_tri_rods.py. And that may be all you need because you are fine having the output go to the output handling of the notebook you run that in.

    However, in your provided command, the developers are sending the output of stdout and stderr all to an .out file. (Typically called a log file and maybe even named with a .log extension in other contexts.) You can recapitulate that, too, using IPython/Jupyter. I'll cover here one way to do the equivalent steps with references to an SO answer with more. In the first cell you'd have:

    %%capture out
    %run mpb_tri_rods.py
    

    Then in the next cell:

    %store out.stdout >tri-rods.out
    %store out.stderr >>tri-rods.out
    

    The first step sends the output of the entire cell to out object of type IPython.utils.capture.CapturedIO. The entire cell is only running the script you invoke with %run.

    The cell I suggest running after that then stores the output from the stdout and stderr channels to a the .out file.

    (I know that seems clunkier but the emphasis in IPython and Jupyter (with a Python kernel) is on interactivity and Pythonic ways. When you consider how to deal with the output below, you may see though it looks more clunky, it is more flexible for working with the output further in a notebook because it is an in memory object if you use those ways and not the shell-only ways, which would require reading the result back in again.)

    This though would produce different output than what your original command would because that would resulting in putting the output from the two channels sequentially whereas !python mpb_tri_rods.py >& tri-rods.out would intermingle the output streams.

    A lot of what you are asking about with using Jupyter to run Python scripts and control stdout & stderr redirection in the more typical IPython/Jupyter ways is covered on StackOverflow already if you know where to look, such as here(and in the several other answers there) and here and elsewhere.

    "Further work on them"...

    You tacked on the phrase "and further work on them". That kind of goes beyond what you asked in your title and normally you are requested on StackOverflow to limit each post to one question. But I'll add some tips since command line vs. Jupyter and using each interchangeably when you are working with something expected to not have an interactive session can be a bit tricky to navigate at the start...

    There's various ways you can view the resulting .out in Jupyter. If you just need to see the beginning or end to see if it was made, you can do !head tri-rods.out or !tail tri-rods.out respectively. !cat tri-rods.out will show you the entire thing inside your notebook. (Because there is no cat command in Jupyter and it is commonly used, you can use without the exclamation point and auto-magics will still handle it as shell command like if you wrote %cat tri-rods.out. However, not all shell command have equivalents that can get handled that way. For instance, tail tri-rods.out won't work in a cell, you need !tail tri-rods.out. I don't know why the inconsistencies. (It probably happens to do with difficulties keeping things working everywhere & discouraging/encouraging certain patterns.)

    And because with use of the IPython/Jupyter ways you'd have the result already as a string, so you can parse it right in the notebook using code.