Hello Stack Overflow community,
I'm currently facing a very perplexing issue with my Jupyter notebooks (.ipynb files) and would greatly appreciate any insights or solutions you might have.
My notebooks are primarily for working with Large Language Models (LLMs) from Hugging Face. They execute perfectly without any issues in Google Colab and also load and run flawlessly in my local Anaconda Jupyter environment.
However, I'm encountering significant problems when trying to use or share these notebooks on other platforms:
On Kaggle: I frequently get "error" messages or general instability, and I suspect it might be related to the same underlying issue as described below.
On GitHub: When I push my notebooks to GitHub, they fail to render, displaying the following error:
There was an error rendering your Notebook: the 'state' key is missing from 'metadata.widgets'. Add 'state' to each, or remove 'metadata.widgets'. Using nbformat v5.10.4 and nbconvert v7.16.6
he crucial and most confusing part is that I am explicitly not using any interactive widgets (like ipywidgets) in these notebooks. The error message points directly to a missing 'state' key within metadata.widgets, but since I'm not incorporating any interactive elements, this section of the metadata shouldn't even be relevant or populated in a problematic way.
I've already tried the standard troubleshooting step of clearing all outputs (Kernel -> Restart & Clear Output in Jupyter/Kaggle) before saving and pushing to GitHub, but unfortunately, the problem persists, and the notebooks still fail to render or upload correctly.
Has anyone encountered this specific metadata.widgets error when not using widgets, especially with notebooks involving LLMs or on platforms like Kaggle and GitHub? Is there a known workaround or a way to rigorously clean this metadata section to ensure proper rendering and upload?
Any suggestions or pointers would be immensely helpful!
Thank you in advance for your time and expertise.
This is a known issue, see discussion here in GitHhub community issue entitled 'Notebook no longer rendering on GitHub: "state" key missing from metadata.widgets' with some suggested solutions, some of which are further expanded on in this helpful resource 'Handling GitHub Invalid Notebooks' that is linked there.
Although it seems it is intermittent and so time may be all that is needed per here:
"This seems to be a transient issue that does resolve itself over time . Check the previous discussion thread regarding same issue that keeps happening time to time. check the response from GH Support on this matter, there are few workarounds suggested in this link.
jupyter/notebook#3555](https://github.com/jupyter/notebook/issues/3555) jupyter/notebook#3555 (comment)"
I'll add that in a lot of cases you'll get better abilities using the historical provided solution and not a quick-preview mode GitHub provides that is meant for developers.
nbviewer is the Jupyter community-provided solution for viewing and sharing notebooks 'static' form. I put 'static' in quotes because a lot of javascript-based features, like Plotly plots, work in nbviewer's renderings. Plus, the nbviewer renderings aren't embedded inside a GitHub page and thus display with less cruft surrounding them are more inviting for those who aren't developers and better for sharing.
The view at GitHub is nothing more than a quick-preview meant for developers. It doesn't support a lot of features of notebooks, such as interactive Plotly plots, and has faulty rendering in a number of ways. Up until a few years ago, GitHub didn't even try rendering the notebook if it had more than a few cells of code. That was better in the long run because Jupyter users realized more quickly they needed to point nbviewer at the GitHub page to see the notebook reliably. I wish their various current 'invalid'-like notes mentioned it still.
More examples of supported features nbviewer supports in 'static' notebooks:
nbsanity.com. (Source code at https://github.com/hamelsmu/nbsanity .)
"I WAS SO FRUSTRATED with the way @github renders Jupyter Notebooks, esp on mobile (You cannot horizontal scroll code, even with nbviewer). It's different than nbviewer because it uses @quarto_pub as the renderer, which has a more sane way of showing notebooks on different platforms.The upshot is, it will also respect all the quarto directives if that happens to be in your notebook" - Source: Hamel Husain (here and here)
Further updates and justification of nbsanity AND NEW AMAZING FEATURES: https://x.com/HamelHusain/status/1869609196486017085 December 2024
"The most annoying gap in AI infra is poor rendering of notebooks on GitHub. Jupyter notebooks are the perfect microblogging medium for technical content, yet are underserved. This is why we are excited to announce nbsanity (link below)" "We've experimented with making notebook publication more accessible with projects like nbdev and fastpages. But something more lightweight is needed. We've drawn inspiration from @simonw 's TIL's. Now we can write TILs with notebooks!!! https://answer.ai/posts/2024-12-13-nbsanity.html"