pythoniterable

What exactly does "iterable" mean in Python? Why isn't my object which implements `__getitem__()` an iterable?

First I want to clarify, I'm NOT asking what is "iterator".

This is how the term "iterable" is defined in Python's doc:

iterable

An object capable of returning its members one at a time. Examples of iterables include all sequence types (such as list, str, and tuple) and some non-sequence types like dict, file objects, and objects of any classes you define with an __iter__() or __getitem__() method.

Iterables can be used in a for loop and in many other places where a sequence is needed (zip(), map(), ...). When an iterable object is passed as an argument to the built-in function iter(), it returns an iterator for the object. This iterator is good for one pass over the set of values. When using iterables, it is usually not necessary to call iter() or deal with iterator objects yourself. The for statement does that automatically for you, creating a temporary unnamed variable to hold the iterator for the duration of the loop.

See also iterator, sequence, and generator.

As other people suggested, using isinstance(e, collections.Iterable) is the most pythonic way to check if an object is iterable.
So I did some test with Python 3.4.3:

from collections.abc import Iterable

class MyTrain:
    def __getitem__(self, index):
        if index > 3:
            raise IndexError("that's enough!")

        return index

for name in MyTrain():
    print(name)  # 0, 1, 2, 3

print(isinstance(MyTrain(), Iterable))  # False

The result is quite strange: MyTrain has defined __getitem__ method, but it is not considered as an iterable object, not to mention it's capable of returning one number at a time.

Then I removed __getitem__ and added the __iter__ method:

from collections.abc import Iterable

class MyTrain:    
    def __iter__(self):
        print("__iter__ called")
        pass

print(isinstance(MyTrain(), Iterable))  # True

for name in MyTrain():
    print(name)  # TypeError: iter() returned non-iterator of type 'NoneType'

It is now considered as a "true" iterable object in spite of it cannot produce anything while iterating.

So did I misunderstand something or is the documentation incorrect?

Solution

  • I think the point of confusion here is that, although implementing __getitem__ does allow you to iterate over an object, it isn't part of the interface defined by Iterable.

    The abstract base classes allow a form of virtual subclassing, where classes that implement the specified methods (in the case of Iterable, only __iter__) are considered by isinstance and issubclass to be subclasses of the ABCs even if they don't explicitly inherit from them. It doesn't check whether the method implementation actually works, though, just whether or not it's provided.

    For more information, see PEP-3119, which introduced ABCs.

    using isinstance(e, collections.Iterable) is the most pythonic way to check if an object is iterable

    I disagree; I would use duck-typing and just attempt to iterate over the object. If the object isn't iterable a TypeError will be raised, which you can catch in your function if you want to deal with non-iterable inputs, or allow to percolate up to the caller if not. This completely side-steps how the object has decided to implement iteration, and just finds out whether or not it does at the most appropriate time.

    As the ABC documentation mentions:

    Checking isinstance(obj, Iterable) detects classes that are registered as Iterable or that have an __iter__() method, but it does not detect classes that iterate with the __getitem__() method. The only reliable way to determine whether an object is iterable is to call iter(obj).

    To add a little more, I think the docs you've quoted are slightly misleading. To quote the iter docs, which perhaps clear this up:

    object must be a collection object which supports the iteration protocol (the __iter__() method), or it must support the sequence protocol (the __getitem__() method with integer arguments starting at 0).

    This makes it clear that, although both protocols make the object iterable, only one is the actual "iteration protocol", and it is this that isinstance(thing, Iterable) tests for. Therefore we could conclude that one way to check for "things you can iterate over" in the most general case would be:

    isinstance(thing, (Iterable, Sequence))

    However, this doesn't work because:

    1. your class doesn't implement __len__ (and doesn't need to anyway, being iterable only requires a sequence-compatible __getitem__); and

    2. Sequence is a complex interface:

      Complex interfaces do not support [virtual subclassing] because an interface is more than just the presence of method names. Interfaces specify semantics and relationships between methods that cannot be inferred solely from the presence of specific method names.

      so even a class that implements both __getitem__ and __len__ doesn't automatically become a Sequence, it needs to explicitly extend the ABC (at which point it would inherit an __iter__ implementation and become Iterable anyway).