pysparkjupyter-notebook

What is the best way to suppress the spark output in the Jupyter notebook?

enter image description here

I recently changed my os from ubuntu to mac.(after 4+ years)

one thing that annoys me a lot is. The spark stdout/stderr now prints on the notebook, rather than staying only inside the console (as on my ubuntu).

I have tried to disable warnings on ipython startup but it didn't work. ~/.ipython/profile_default/startup/disable-warning.py

import warnings
warnings.filterwarnings('ignore')

on the other hand, I am not entirely sure if that is mac problem. Is something went stupidly wrong during my dev env setup?

Thank you!

Solution

  • The modification of the notebook output comes from one of the changes made in the release 6.0.0 of ipykernel.

    All outputs to stdout/stderr should now be captured, including subprocesses and output of compiled libraries (blas, lapack....). In notebook server, some outputs that would previously go to the notebooks logs will now both head to notebook logs and in notebooks outputs.

    A subsequent fix provides a way to restore the previous behavior. The fix consists in disabling the capture new behavior by turning it off through the capture_fd_output flag, see this comment for more detail.

    You can configure it by turning it off in your ipython profile.

    # create a default profile
ipython profile create
# turn off the capture flag
echo "c.IPKernelApp.capture_fd_output = False" >> \
     "~/.ipython/profile_default/ipython_kernel_config.py"

    That's it ! All the outputs from Java, Spark and Ivy will no more be displayed in the notebook but only in the logs.

    Notes