I'm experimenting with metaprogramming in Python (CPython 3.10.13) and noticed some weird behavior with object.__new__ (well, weird to me, at least). Take a look at the following experiment (not practical code, just an experiment) and the comments. Note that object.__new__ seems to change it's behavior based on the first argument:
# Empty class inherit __new__ and __init__ from object
class Empty:
pass
# Confirmation of inheritance
assert Empty.__new__ is object.__new__, "Different __new__"
assert Empty.__init__ is object.__init__, "Different __init__"
empty_obj = Empty()
uinit_empty_obj = object.__new__(Empty)
assert type(empty_obj) is type(uinit_empty_obj), "Different types"
try:
object.__new__(Empty, 10, 'hi', hello='bye')
except TypeError as e:
# repr(e) mentioned the Empty class
print(repr(e))
# Overwrite the object __new__ and __init__ methods
# __new__ and __init__ with the same signature
class Person:
def __new__(cls, name, age):
"""Does nothing bassicaly. Just overwrite `object.__new__`."""
print(f'Inside {cls.__name__}.__new__')
return super().__new__(cls)
def __init__(self, name, age):
print(f'Inside {type(self).__name__}.__init__')
self.name = name
self.age = age
a_person = Person('John Doe', 25)
uinit_person = Person.__new__(Person, 'Michael', 40)
try:
# Seems an obvious error since object() doesn't take any arguments
another_uinit_person = object.__new__(Person, 'Ryan', 25)
except TypeError as e:
# Indeed raises TypeError, but now there isn't a mention of the Person class in repr(e)
print('`another_uinit_person` :', repr(e))
# Now, some weird things happen (well, weird for me).
# Inherit __new__ from object and overwrite __init__.
# __new__ and __init__ with unmatching signatures.
# A basic Python class. Works just fine like suppose to.
class Vehicle:
def __init__(self, model):
self.model = model
# Confirmation of __new__ inheritance.
assert Vehicle.__new__ is object.__new__, "Nop, it isn't"
a_vehicle = Vehicle('Honda')
# I would understand if CPython autogenerated a __new__ method matching __init__
# or a __new__ method that accepts all arguments.
# The following try-except-else suggests the last, but the assert statement above
# indicates that Vehicle.__new__ is actually object.__new__.
try:
# Doesn't raise any exceptions
uinit_vehicle = Vehicle.__new__(Vehicle, 'Honda', 10, ('four-wheels',), hello='bye')
except Exception as e:
print(repr(e))
else:
print('`uinit_vehicle` : constructed just fine', uinit_vehicle)
# Now the following runs just fine
try:
# Doesn't raise any exceptions
another_unit_vehicle = object.__new__(Vehicle, 'Toyota')
another_unit_vehicle = object.__new__(Vehicle, 'Toyota', 100, four_wheels=True)
except Exception as e:
print(repr(e))
else:
print('`another_unit_vehicle` : constructed just fine:', another_unit_vehicle)
I got the following output:
TypeError('Empty() takes no arguments')
Inside Person.__new__
Inside Person.__init__
Inside Person.__new__
`another_uinit_person` : TypeError('object.__new__() takes exactly one argument (the type to instantiate)')
`uinit_vehicle` : constructed just fine <__main__.Vehicle object at 0x00000244D15A7A90>
`another_unit_vehicle` : constructed just fine: <__main__.Vehicle object at 0x00000244D15A7A30>
My questions:
TypeError mentioned the Empty class and the second just object.__new__?object.__new__(Person, 'Ryan', 25) raised TypeError and object.__new__(Vehicle, 'Toyota') and object.__new__(Vehicle, 'Toyota', 100, four_wheels=True) didn't?Basically: what object.__new__ does under the hood?
It seems to me that it is performing a somewhat weird check on the first argument's __new__ and/or __init__ override methods, if any.
Python's object.__init__ and object.__new__ base methods suppress errors about excess arguments in the common situation where exactly one of them has been overridden, and the other has not. The non-overriden method will ignore the extra arguments, since they usually get passed in automatically (rather than by an explicit call to __new__ or __init__ where the programmer should know better).
That is, neither of these classes will cause issues in the methods they inherit:
class OnlyNew:
def __new__(self, *args):
pass
# __init__ is inherited from object
class OnlyInit:
def __init__(self, *args):
pass
# __new__ is inherited from object
# tests:
object.__new__(OnlyInit, 1, 2, 3, 4) # no error
object.__init__(object.__new__(OnlyNew), 1, 2,3, 4) # also no error
However, when you override one of the methods, you must avoid excess arguments when you call the base class version of the method you overrode.
# bad tests:
try:
object.__new__(OnlyNew, 1, 2, 3, 4)
except Exception as e:
print(e) # object.__new__() takes exactly one argument (the type to instantiate)
try:
object.__init__(object.__new__(OnlyInit), 1, 2, 3, 4)
except Exception as e:
print(e) # object.__init__() takes exactly one argument (the type to instantiate)
Furthermore if you override both __new__ and __init__, you need to both of call the base class methods with no extra arguments, since you should know what you're doing if you're implementing both methods.
class OverrideBoth:
def __new__(self, *args):
pass
def __init__(self, *args):
pass
# more bad tests, object has zero tolerance for extra arguments in this situation
try:
object.__new__(OverrideBoth, 1, 2, 3, 4)
except Exception as e:
print(e) # object.__new__() takes exactly one argument (the type to instantiate)
try:
object.__init__(object.__new__(OverrideBoth), 1, 2,3, 4)
except Exception as e:
print(e) # object.__init__() takes exactly one argument (the instance to initialize)
You can see the implementation of these checks in the CPython source code. Even if you don't know C very well, it's pretty clear what it's doing. There's a different code path that handles classes like your Empty that don't override either method (which is why that exception message is a bit different).