Besides the obvious asking "again" about __new__ and __init__ in Python - I can ensure, I know what it does. I'll demonstrate some strange and to my opinion undocumented behavior, for which I seek professional help :).
I'm implementing several features like abstract methods, abstract classes, must-override methods, singletone behavior, slotted classes (automatic inference of __slots__) and mixin classes (deferred slots) using a user-defined meta-class called ExtendedType. The following code can be found as a whole at pyTooling/pyTooling on the development branch.
Thus, the presented question is a stripdown and simplified variant demonstrating the strange behavior of object.__new__.
Depending on the internal algorithms of ExtendedType, it might decide a class A is abstract. If so, the __new__ method is replaced by a dummy method raising an exception (AbstractClassError). Later, when a class B(A) inherits from A, the meta-class might come to the decision, B isn't abstract anymore, thus we want to allow the object creation again and allow calling for the original __new__ method. Therefore, the original method is preserved as a field in the class.
To simplify the internal algorithms for the abstractness decision, the meta-class implements a boolean named-parameter abstract.
class AbstractClassError(Exception):
pass
class M(type):
# staticmethod
def __new__(cls, className, baseClasses, members, abstract):
newClass = type.__new__(cls, className, baseClasses, members)
if abstract:
def newnew(cls, *_, **__):
raise AbstractClassError(f"Class is abstract")
# keep original __new__ and exchange it with a dummy method throwing an error
newClass.__new_orig__ = newClass.__new__
newClass.__new__ = newnew
else:
# 1. replacing __new__ with original (preserved) method doesn't work
newClass.__new__ = newClass.__new_orig__
return newClass
class A(metaclass=M, abstract=True):
pass
class B(A, abstract=False):
def __init__(self, arg):
self.arg = arg
b = B(5)
When instantiating B we'll try two cases:
b = B(5)TypeError: object.__new__() takes exactly one argument (the type to instantiate)
b = B()TypeError: B.__init__() missing 1 required positional argument: 'arg'
The error message of the latter case is expected, because __init__ of B expects an argument arg. The strange behavior is in case 1, where it reports object.__new__() takes no additional parameters except of the type.
So let's investigate if swapping methods worked correctly:
print("object.__new__ ", object.__new__)
print("A.__new_orig__ ", A.__new_orig__)
print("A.__new__ ", A.__new__)
print("B.__new__ ", B.__new__)
Results:
object.__new__ <built-in method __new__ of type object at 0x00007FFE30EDD0C0>
A.__new_orig__ <built-in method __new__ of type object at 0x00007FFE30EDD0C0>
A.__new__ <function M.__new__.<locals>.newnew at 0x000001CF11AE5A80>
B.__new__ <built-in method __new__ of type object at 0x00007FFE30EDD0C0>
So, the preserved method in __new_orig__ is identical to object.__new__ and is again the same after swapping back the __new__ method in class B.
Let's take two classes X and Y(X) and instantiate them:
class X:
pass
class Y(X):
def __init__(self, arg):
self.arg = arg
y = Y(3)
Of cause this will work, but are the __new__ methods different?
object.__new__ <built-in method __new__ of type object at 0x00007FFE3B61D0C0>
A.__new_orig__ <built-in method __new__ of type object at 0x00007FFE3B61D0C0>
A.__new__ <function M.__new__.<locals>.newnew at 0x000001CD1FB459E0>
B.__new__ <built-in method __new__ of type object at 0x00007FFE3B61D0C0>
X.__new__ <built-in method __new__ of type object at 0x00007FFE3B61D0C0>
Y.__new__ <built-in method __new__ of type object at 0x00007FFE3B61D0C0>
Also X and Y use the same __new__ method as B or object.
So let's instantiate Y and B and compare results:
print("Y.__new__ ", Y.__new__)
y = Y(3)
print("y.arg ", y.arg)
print("B.__new__ ", B.__new__)
b = B(5)
print("b.arg ", y.arg)
Results:
Y.__new__ <built-in method __new__ of type object at 0x00007FFE3B61D0C0>
y.arg 3
B.__new__ <built-in method __new__ of type object at 0x00007FFE3B61D0C0>
Traceback (most recent call last):
File "C:\Temp\newIstKomisch.py", line 67, in <module>
b = B(5)
^^^^
TypeError: object.__new__() takes exactly one argument (the type to instantiate)
Question 1: Why does new accept parameters for Y, but not for B?
When an object is created, the __call__ method of the meta-class is executed, which roughly translates to:
class M(type):
...
def __call__(cls, *args, **kwargs):
inst = cls.__new__(cls, *args, **kwargs)
inst.__init__(*args, **kwargs)
return inst
It first calls __new__ to create an instance and then it calls __init__ to initialize the object. One might argue and say: "maybe there is magic behavior in call" to check if a build-in or user-defined method is called"...
Let's quickly check how object.__new__ behaves:
o = object.__new__(object, 1)
Result:
TypeError: object() takes no arguments
Observation: The error message is different then what we got before. This says "no arguments", the other says "exactly one argument".
Alternatively, we can create an object by hand skipping the meta-class:
y = Y.__new__(Y, 3)
print("Y.__new__(Y, 3) ", y)
y.__init__(3)
print("y.__init__(3) ", y.arg)
Result:
Y.__new__(Y, 3) <__main__.Y object at 0x0000020ED770BD40>
y.__init__(3) 3
Here we clearly see __new__ can accept additional parameters and ignore them.
So let's compare to manual instance creation of B:
b = B.__new__(B, 5)
print("B.__new__(B, 5) ", b)
b.__init__(5)
print("b.__init__(5) ", b.arg)
Result:
Traceback (most recent call last):
File "C:\Temp\newIstKomisch.py", line 51, in <module>
b = B.__new__(B, 5)
^^^^^^^^^^^^^^^
TypeError: object.__new__() takes exactly one argument (the type to instantiate)
Question 2: How can the same method have different behavior and exception handling?
Additional notes:
M.__new__ or swapped XXX.__new__ methods instead of M.__call__, so the object creation time isn't influenced. Modifying the meta-classes call would have a huge performance impact.Attachements:
That is sure a lot of research for a question.
But the answer is more simple:
objects __new__ and __init__ simply special case the "forgiveness of extra arguments" in a way that it feels natural to create new classes with a custom __init__ method, with no need to fiddle with __new__.
So, in short, object new checks if the class it is instantiating have a custom __init__ and no custom __new__ - if so, it "forgives" extra args and kwargs. And object's default __init__ does the converse: it checks if the class it is "initting" have a custom __new__ and no custom __init__. If so it also forgives (and forgets) about any extra parameters. The "custom" verification here simply checks if there is a __new__ method present in any class' dict in the __mro__ - so that even setting the same object.__new__ class in a subclass won't work.
This strange-sounding special case is needed, and is baked in since a long-time in Python, because without it, whenever creating a class with a __init__ method taking arguments, without implementing also a __new__ method that would fail - so the case for "simplify class customization" by having a simpler __init__ method rather than modifying __new__ would be moot.
Here are a few examples on the REPL that make that clear:
In [11]: class A(object): pass
In [12]: b = A.__new__(A, 3)
TypeError (...)
TypeError: A() takes no arguments
# There is no custom `__init__`, so it fails
In [13]: class A(object):
...: def __init__(self, *args):
...: pass
...:
In [14]: b = A.__new__(A, 3)
# There was a custom `__init__` so, object.__new__ forgives us.
# and finally your case, both a __new__ and __init__ even if `cls.__new__ is object.__new__` is true, errors:
In [17]: class A(object):
...: def __new__(cls, *args):
...: raise NotImplementedError()
...: def __init__(self, *args):
...: pass
...:
In [18]: class B(A):
...: __new__ = object.__new__
...:
In [19]: c = B() #<- works with no args
In [20]: c = B(3)
TypeError
TypeError: object.__new__() takes exactly one argument (the type to instantiate)
# And just one example with __init__ to show the converse case:
In [28]: class A(object):
...: def __new__(cls, *args):
...: # I strip down the extra args:
...: return super().__new__(cls)
...: # no custom __init__
...:
In [29]: b = A(3) # <- works
In [30]: class A(object):
...: def __new__(cls, *args):
...: # I strip down the extra args:
...: return super().__new__(cls)
...: # with a custom __init__ forwarding extra args
...: def __init__(self, *args):
...: print("init")
...: super().__init__(*args)
...:
In [31]: b = A(3) # <- errors out
init
TypeError (...)
Cell In[30], line 8, in A.__init__(self, *args)
6 def __init__(self, *args):
7 print("init")
----> 8 super().__init__(*args)
TypeError: object.__init__() takes exactly one argument (the instance to initialize)
for your case, you can't simply restore object.__new__ in a custom "granddaughter" class - you will need instead to check if __orig_new__ is object.__new__ and if so, use a custom __new__ that will strip extra arguments before calling object.__new__.