pythonobjectmetaprogramming

Behavior of object.__new__ Python dunder. What is happening under the hood?


I'm experimenting with metaprogramming in Python (CPython 3.10.13) and noticed some weird behavior with object.__new__ (well, weird to me, at least). Take a look at the following experiment (not practical code, just an experiment) and the comments. Note that object.__new__ seems to change it's behavior based on the first argument:

# Empty class inherit __new__ and __init__ from object
class Empty:
    pass

# Confirmation of inheritance
assert Empty.__new__ is object.__new__, "Different __new__"
assert Empty.__init__ is object.__init__, "Different __init__"

empty_obj = Empty()
uinit_empty_obj = object.__new__(Empty)

assert type(empty_obj) is type(uinit_empty_obj), "Different types"

try:
    object.__new__(Empty, 10, 'hi', hello='bye')
except TypeError as e:
    # repr(e) mentioned the Empty class
    print(repr(e))

# Overwrite the object __new__ and __init__ methods
# __new__ and __init__ with the same signature
class Person:
    def __new__(cls, name, age):
        """Does nothing bassicaly. Just overwrite `object.__new__`."""
        print(f'Inside {cls.__name__}.__new__')
        return super().__new__(cls)
    
    def __init__(self, name, age):
        print(f'Inside {type(self).__name__}.__init__')
        self.name = name
        self.age = age

a_person = Person('John Doe', 25)
uinit_person = Person.__new__(Person, 'Michael', 40)

try:
    # Seems an obvious error since object() doesn't take any arguments
    another_uinit_person = object.__new__(Person, 'Ryan', 25)
except TypeError as e:
    # Indeed raises TypeError, but now there isn't a mention of the Person class in repr(e)
    print('`another_uinit_person` :', repr(e))

# Now, some weird things happen (well, weird for me).
# Inherit __new__ from object and overwrite __init__.
# __new__ and __init__ with unmatching signatures.
# A basic Python class. Works just fine like suppose to.
class Vehicle:
    def __init__(self, model):
        self.model = model

# Confirmation of __new__ inheritance.
assert Vehicle.__new__ is object.__new__, "Nop, it isn't"

a_vehicle = Vehicle('Honda')

# I would understand if CPython autogenerated a __new__ method matching __init__
# or a __new__ method that accepts all arguments.
# The following try-except-else suggests the last, but the assert statement above 
# indicates that Vehicle.__new__ is actually object.__new__.
try:
    # Doesn't raise any exceptions
    uinit_vehicle = Vehicle.__new__(Vehicle, 'Honda', 10, ('four-wheels',), hello='bye')
except Exception as e:
    print(repr(e))
else:
    print('`uinit_vehicle` : constructed just fine', uinit_vehicle)

# Now the following runs just fine
try:
    # Doesn't raise any exceptions
    another_unit_vehicle = object.__new__(Vehicle, 'Toyota')
    another_unit_vehicle = object.__new__(Vehicle, 'Toyota', 100, four_wheels=True)
except Exception as e:
    print(repr(e))
else:
    print('`another_unit_vehicle` : constructed just fine:', another_unit_vehicle)

I got the following output:

TypeError('Empty() takes no arguments')
Inside Person.__new__
Inside Person.__init__
Inside Person.__new__
`another_uinit_person` : TypeError('object.__new__() takes exactly one argument (the type to instantiate)')
`uinit_vehicle` : constructed just fine <__main__.Vehicle object at 0x00000244D15A7A90>
`another_unit_vehicle` : constructed just fine: <__main__.Vehicle object at 0x00000244D15A7A30>

My questions:

  1. Why the first TypeError mentioned the Empty class and the second just object.__new__?
  2. Why object.__new__(Person, 'Ryan', 25) raised TypeError and object.__new__(Vehicle, 'Toyota') and object.__new__(Vehicle, 'Toyota', 100, four_wheels=True) didn't?

Basically: what object.__new__ does under the hood?

It seems to me that it is performing a somewhat weird check on the first argument's __new__ and/or __init__ override methods, if any.


Solution

  • Python's object.__init__ and object.__new__ base methods suppress errors about excess arguments in the common situation where exactly one of them has been overridden, and the other has not. The non-overriden method will ignore the extra arguments, since they usually get passed in automatically (rather than by an explicit call to __new__ or __init__ where the programmer should know better).

    That is, neither of these classes will cause issues in the methods they inherit:

    class OnlyNew:
        def __new__(self, *args):
            pass
    
        # __init__ is inherited from object
    
    class OnlyInit:
        def __init__(self, *args):
            pass
    
        # __new__ is inherited from object
    
    # tests:
    object.__new__(OnlyInit, 1, 2, 3, 4)                  # no error
    object.__init__(object.__new__(OnlyNew), 1, 2,3, 4)   # also no error
    

    However, when you override one of the methods, you must avoid excess arguments when you call the base class version of the method you overrode.

    # bad tests:
    try:
        object.__new__(OnlyNew, 1, 2, 3, 4)
    except Exception as e:
        print(e) # object.__new__() takes exactly one argument (the type to instantiate)
    try:
        object.__init__(object.__new__(OnlyInit), 1, 2, 3, 4)
    except Exception as e:
        print(e) # object.__init__() takes exactly one argument (the type to instantiate)
    

    Furthermore if you override both __new__ and __init__, you need to both of call the base class methods with no extra arguments, since you should know what you're doing if you're implementing both methods.

    class OverrideBoth:
        def __new__(self, *args):
            pass
    
        def __init__(self, *args):
            pass
    
    # more bad tests, object has zero tolerance for extra arguments in this situation
    try:
        object.__new__(OverrideBoth, 1, 2, 3, 4)
    except Exception as e:
        print(e) # object.__new__() takes exactly one argument (the type to instantiate)
    
    try:
        object.__init__(object.__new__(OverrideBoth), 1, 2,3, 4)
    except Exception as e:
        print(e) # object.__init__() takes exactly one argument (the instance to initialize)
    

    You can see the implementation of these checks in the CPython source code. Even if you don't know C very well, it's pretty clear what it's doing. There's a different code path that handles classes like your Empty that don't override either method (which is why that exception message is a bit different).