pythoninitializationoverloadingfunctoolssingle-dispatch

Why does overloading __new__ with singledispatchmethod not work as expected?


I want my __new__ method to behave differently in some cases and wanted to split it into overloaded functions with singledispatchmethod.
For other methods it works like expected. However for __new__ it does not work, the overloading functions are never called. What is the reason for that?

from functools import singledispatchmethod

class Foo:
     @singledispatchmethod
     def __new__(cls, arg1, **kwargs):
         return "default call"

     @__new__.register(int)
     def _(cls, arg1, **kwargs):
         return "called with int " + str(arg1)

print(Foo("hi"))
# default call
print(Foo(1))
# default call

As an experiment also used singledispatch instead but without success.


Solution

  • You might think that singledispatchmethod is just a version of singledispatch that dispatches based on the second argument instead of the first, but that's not how it works.

    Instead, it's written as a class that implements the descriptor protocol to customize attribute access. When you access a singledispatchmethod-decorated method, the attribute access returns a closure object that dispatches based on the first argument to that closure object.

    So for example, if a class named Example had a singledispatchmethod named sdm, then Example().sdm(1) would dispatch on the type of 1, but Example.sdm(Example(), 1) would dispatch on the type of Example()!


    __new__ isn't a regular method. It's supposed to be a staticmethod, or at least, it's supposed to be something that behaves like a staticmethod when accessed. (Ordinarily, type.__new__ would automatically convert __new__ methods to staticmethods, but it only does that when __new__ is an ordinary Python function object, and as mentioned, singledispatchmethod is implemented as a custom class.)

    Particularly, when you do Foo(1), the resulting __new__ invocation works like Foo.__new__(Foo, 1). It retrieves the __new__ attribute on Foo, then calls whatever it finds with Foo and 1 as arguments.

    Due to the way singledispatchmethod performs dispatch, this dispatches based on the type of Foo, not the type of 1.


    Ordinarily, if you wanted staticmethod-like behavior from singledispatchmethod, the way to get it would be to take advantage of a singledispatchmethod feature that lets it wrap other decorators, like so:

    # This doesn't do what you need.
    
    @functools.singledispatchmethod
    @staticmethod
    def __new__(cls, arg1, **kwargs):
        ...
    
    @__new__.register(int)
    @staticmethod
    def _(cls, arg1, **kwargs):
        ...
    

    However, this doesn't do anything to fix the problem of which argument we're dispatching on.

    Instead, you could write __new__ as a regular method that delegates to a singledispatch helper, and reorder the helper's arguments so the argument you want to dispatch on is in front:

    import functools
    
    class Foo:
        def __new__(cls, arg1, **kwargs):
            return _new_helper(arg1, cls, **kwargs)
    
    @functools.singledispatch
    def _new_helper(arg1, cls, **kwargs):
        return "default call"
    
    @_new_helper.register(int)
    def _(arg1, cls, **kwargs):
        return "called with int " + str(arg1)
    

    Or, of course, you could ditch the whole singledispatch thing and just write the dispatch handling yourself:

    import functools
    
    class Foo:
        def __new__(cls, arg1, **kwargs):
            if isinstance(arg1, int):
                return "called with int " + str(arg1)
            else:
                return "default call"