pythonjupyter-notebook

Organize/run python notebook cells (or functions) as a flow diagram/ mermaid chart


I use python as part of my daily workflow mostly for modeling and data analysis and I've been dying to use some system similar to the one I outlined in this picture (here I am using obsidian canvas as an example). My dream is to have something like this with one click snapshot to save current cell layouts and code for quick and dirty version control with simple notes. Are there any existing systems that come close to this? I have not messed with apache airflow but seems to be similar but much more production oriented instead of proof of concept/prototyping oriented

ideal software would

The idea here is that I have to quickly experiment with so many variations of data, filtering, modeling approaches, dataset validations etc. So I want to remember what has worked so far without worrying about making perfect/permanent code. Once proof of concept is established I move from notebooks to python package etc.

enter image description here


Solution

  • https://link.makinarocks.ai/

    I found this jupyter lab extension that works fairly well!! caches each node and can combine nodes, pretty great! the DAG is not intractable as far as I know though but still a great solution.

    note only works with up to python 3.9 at this time.

    enter image description here