pythonpython-3.xwindowsgraphvizpygraphviz

Pygraphviz crashes after drawing 170 graphs

I am using pygraphviz to create a large number of graphs for different configurations of data. I have found that no matter what information is put in the graph the program will crash after drawing the 170th graph. There are no error messages generated the program just stops. Is there something that needs to be reset if drawing this many graphs?

I am running Python 3.7 on a Windows 10 machine, Pygraphviz 1.5, and graphviz 2.38

    for graph_number in range(200):
        config_graph = pygraphviz.AGraph(strict=False, directed=False, compound=True, ranksep='0.2', nodesep='0.2')

        # Create Directory
        if not os.path.exists('Graph'):
            os.makedirs('Graph')

        # Draw Graph      
        print('draw_' + str(graph_number))
        config_graph.layout(prog = 'dot')
        config_graph.draw('Graph/'+str(graph_number)+'.png')
Solution

  • I was able to constantly reproduce the behavior with:

    1. Python 3.7.6 (pc064 (64bit), then also with pc032)

    2. PyGraphviz 1.5 (that I built - available for download at [GitHub]: CristiFati/Prebuilt-Binaries - Various software built on various platforms. (under PyGraphviz, naturally).
      Might also want to check [SO]: Installing pygraphviz on Windows 10 64-bit, Python 3.6 (@CristiFati's answer))

    3. Graphviz 2.42.2 ((pc032) same as #2.)

    I suspected an Undefined Behavior somewhere in the code, even if the behavior was precisely the same:

    Did some debugging (added some print(f) statements in agraph.py, and cgraph.dll (write.c)).
    PyGraphviz invokes Graphviz's tools (.exes) for many operations. For that, it uses subprocess.Popen and communicates with the child process via its 3 available streams (stdin, stdout, stderr).

    From the beginning I noticed that 170 * 3 = 510 (awfully close to 512 (0x200)), but didn't pay as much attention as I should have until later (mostly because the Python process (running the code below) had no more than ~150 open handles in Task Manager (TM) and also Process Explorer (PE)).

    However, a bit of Googleing revealed:

    Below is your code that I modified for debugging and reproducing the error. It needs (for code shortness' sake, as same thing can be achieved via CTypes) the PyWin32 package (python -m pip install pywin32).

    code00.py:

    #!/usr/bin/env python

import os
import sys
#import time

import pygraphviz as pgv
import win32file as wfile


def handle_graph(idx, dir_name):
    graph_name = "draw_{:03d}".format(idx)
    graph_args = {
        "name": graph_name,
        "strict": False,
        "directed": False,
        "compound": True,
        "ranksep": "0.2",
        "nodesep": "0.2",
    }
    graph = pgv.AGraph(**graph_args)
    # Draw Graph      
    img_base_name = graph_name + ".png"
    print("  {:s}".format(img_base_name))
    graph.layout(prog="dot")
    img_full_name = os.path.join(dir_name, img_base_name)
    graph.draw(img_full_name)
    graph.close()  # !!! Has NO (visible) effect, but I think it should be called anyway !!!


def main(*argv):
    print("OLD max open files: {:d}".format(wfile._getmaxstdio()))
    # 513 is enough for your original code (170 graphs), but you can set it up to 8192
    #wfile._setmaxstdio(513)  # !!! COMMENT this line to reproduce the crash !!!
    print("NEW max open files: {:d}".format(wfile._getmaxstdio()))

    dir_name = "Graph"
    # Create Directory
    if not os.path.isdir(dir_name):
        os.makedirs(dir_name)

    #ts_global_start = time.time()
    start = 0
    count = 170
    #count = 1
    step_sleep = 0.05
    for i in range(start, start + count):
        #ts_local_start = time.time()
        handle_graph(i, dir_name)
        #print("  Time: {:.3f}".format(time.time() - ts_local_start))
        #time.sleep(step_sleep)
    handle_graph(count, dir_name)
    #print("Global time: {:.3f}".format(time.time() - ts_global_start - step_sleep * count))


if __name__ == "__main__":
    print("Python {:s} {:03d}bit on {:s}\n".format(" ".join(elem.strip() for elem in sys.version.split("\n")),
                                                   64 if sys.maxsize > 0x100000000 else 32, sys.platform))
    rc = main(*sys.argv[1:])
    print("\nDone.\n")
    sys.exit(rc)

    Output:

    e:\Work\Dev\StackOverflow\q060876623> "e:\Work\Dev\VEnvs\py_pc064_03.07.06_test0\Scripts\python.exe" ./code00.py
Python 3.7.6 (tags/v3.7.6:43364a7ae0, Dec 19 2019, 00:42:30) [MSC v.1916 64 bit (AMD64)] 064bit on win32

OLD max open files: 512
NEW max open files: 513
  draw_000.png
  draw_001.png
  draw_002.png

...

  draw_167.png
  draw_168.png
  draw_169.png

Done.

    Conclusions:

    Side notes: