pythonmypypython-typingtypeddict

With type hinting, how do I require a key value pair when the key has an invalid identifier?


With type hinting, how do I require a key value pair in python, where the key is an invalid identifier? By required I mean that the data is required by type hinting and static analysis tools like mypy/pyright + IDES. For context: keys with invalid python identifiers are sent over json.

A dict payload with key value pair requirements must be ingested. See below for the additional requirements.

Here is a sample payload:

{
  'a': 1, # required pair with valid identifier
  '1req': 'blah', #required key with invalid identifier
  'someAdditionalProp': [] # optional additional property, value must be list
}

The requirements for the ingestion of this dict payload are are:

  1. there is a valid identifier 'a', with type int that is required
  2. there is an invalid identifier '1req' with type string that is required
  3. additional properties can be passed in with any string keys different from the above two and value list
  4. all input types must be included so TypedDict will not work because it does not capture item 3
  5. in the real life use case the payload can contain n invalidly named identifiers
  6. in real life there can be other known keys like b: float that are optional and are not the same as item 3 additional properties. They are not the same because the value type is different, and this key is a known literal (like 'b') but item 3 keys are strings but their value is unknown

What I want is a class or function signature that ingests the above payload and meets the above type hinting requirements for all required and optional key value pairs. Errors should be thrown for invalid inputs to the class or function in mypy/an IDE with type checking turned on.

An example of an error that would work is:

Argument of type "tuple[Literal['otherReq'], None]" cannot be assigned to parameter "args" of type "OptionalDictPair" in function "__new__"
  "tuple[Literal['otherReq'], None]" is incompatible with "OptionalDictPair"
    Tuple entry 2 is incorrect type
      Type "None" cannot be assigned to type "list[Unknown]"PylancereportGeneralTypeIssues

A naive implementation that does NOT meet requirements would be:

DictType = typing.Mapping[
    str,
    typing.Union[str, int, list]
]

def func_with_type_hinting(arg: DictType):
    pass

func_with_type_hinting(
    {
        'a': 1,
        '1req': 'blah',
        'someAdditionalProp': None
    }
)  # static analysis should show an error here, someAdditionalProp's type is wrong

Solution

  • Option 1 (frozenset of tuples) is my favorite and meets all requirements. The only caviat is that arguments must be passed in required then optional order.

    Options that I know are:

    1. Use a frozenset of tuples to require that the required args are passed first, then the optional args. This assumes that rather than passing the dict in as a dict instance a frozenset of tuples of key value pairs are passed in instead.
    OptionalDictPair = typing.Tuple[typing_extensions.LiteralString, list]
    
    RequiredDictPairs = typing.Union[
        typing.Tuple[typing_extensions.Literal['a'], int],
        typing.Tuple[typing_extensions.Literal['1req'], int]
    ]
    
    ReqAndOptionalDictPairs = typing.Union[
        typing.Tuple[typing_extensions.Literal['a'], str],
        typing.Tuple[typing_extensions.Literal['1req'], int],
        OptionalDictPair
    ]
    
    
    _T_co = typing.TypeVar("_T_co", covariant=True)
    
    class frozenset_with_length(frozenset[_T_co]):
        def __new__(
                cls,
                required_1: RequiredDictPairs,
                required_2: RequiredDictPairs,
                *args: OptionalDictPair) -> typing.Union[frozenset_with_length[RequiredDictPairs], frozenset_with_length[ReqAndOptionalDictPairs]]:
            req_args = (required_1, required_2)
            if not args:
                return super().__new__(cls, *req_args) # type: ignore
            all_args = tuple(req_args)
            all_args.extend(args)
            return super().__new__(cls, *all_args) # type: ignore
    
    tuple_items = frozenset_with_length(*(
        ('a', 1),
        ('1req', 1),
        ('additionalProExample', []),
    ))
    
    1. use a function and move the invalid arg value into a union in kwargs def some_fun(*, a: int, *kwargs: typing.Union[list, str]):
    1. use a function and use *args to define the values for the invalidly named parameter def some_fun(a: int, *args: str, *kwargs: list):