pythongeneratorabstract-base-class

The Python Generator abstract base class doesn't implement the necessary __del__, is this intentional?


Generator objects in Python are required to have a close method that exists to ensure that context managers are exited and try...finally: blocks are run before the object is garbage collected.

PEP 342 defines the methods send,throw,close and __del__ that generators must implement. Specifically, it states:

g.__del__() is a wrapper for g.close(). This will be called when the generator object is garbage-collected (in CPython, this is when its reference count goes to zero).

The abstract type for Generator is in collections.abc

class Generator(Iterator):

    __slots__ = ()

    def __next__(self):
        """Return the next item from the generator.
        When exhausted, raise StopIteration.
        """
        return self.send(None)

    @abstractmethod
    def send(self, value):
        """Send a value into the generator.
        Return next yielded value or raise StopIteration.
        """
        raise StopIteration

    @abstractmethod
    def throw(self, typ, val=None, tb=None):
        """Raise an exception in the generator.
        Return next yielded value or raise StopIteration.
        """
        if val is None:
            if tb is None:
                raise typ
            val = typ()
        if tb is not None:
            val = val.with_traceback(tb)
        raise val

    def close(self):
        """Raise GeneratorExit inside generator.
        """
        try:
            self.throw(GeneratorExit)
        except (GeneratorExit, StopIteration):
            pass
        else:
            raise RuntimeError("generator ignored GeneratorExit")

    @classmethod
    def __subclasshook__(cls, C):
        if cls is Generator:
            return _check_methods(C, '__iter__', '__next__',
                                  'send', 'throw', 'close')
        return NotImplemented

This abstract type enforces that send, throw, and close are implemented in subclasses, but doesn't implement __del__ either abstractly or concretely, or enforce that it's implemented. Its metaclasses don't either.

Naively, producing a subclass that doesn't manually define a __del__ which wraps close gives Generators which are not correctly cleaned up after. The garbage collector only calls __del__, so if __del__ doesn't exist, close is not called.

Is this intentional?

In a related question, snakecharmerb pointed out to me that __del__ can be fraught to implement, as indicated by the language reference, but I can't understand why that wouldn't also apply to the correct implementation of __del__ as a wrapper for close in Python's native generator objects.


Solution

  • It seems to be intentional. In an issue on the python bugtracker discussing this, Guido said that

    The presence of a __del__ method can cause subtle behavior changes to the GC, so I worry that adding __del__ to that class now is going to break currently-working code.

    Let's not destabilize the Generator class.