pythontypesmypybottle

Python mypy type checking not working as expected


I'm new to python, and am a huge fan of static type checkers. I have some code that handles file uploads with the Bottle framework. See below.

def transcribe_upload(upload: FileUpload) -> Alternative:
  audio:AudioSource = upload_source(upload)
  ...

def upload_source(upload:FileUpload) -> AudioSource:
  ...

I made a really simple mistake and passed upload.file (the file-like object) to upload_source instead of the entire FileUpload object.

def transcribe_upload(upload: FileUpload) -> Alternative:
  audio:AudioSource = upload_source(upload.file) # This is incorrect!

The typechecker didn't catch it. In fact, it doesn't catch ANY incorrect parameter passing to upload_source:

def transcribe_upload(upload: FileUpload) -> Alternative:
  audio:AudioSource = upload_source(4)           # Why isn't mypy giving me an error?
  audio:AudioSource = upload_source(upload.asdf) # Why isn't mypy giving me an error?

What's going on? I tested some basic functions separately and the typechecker caught when I tried to pass a number to a function that wanted a string, and it worked. What am I missing here?

EDIT

@kojiro suggsted that FileUpload is equivalent to Any. I think that may be correct. Here is the source for FileUpload. It's imported like this: from bottle import FileUpload.

If that's the case, why is it letting me use FileUpload as if it were a type? (If I mess up the name, like FileUpld it does give me an error).

More importantly, how do I get real types? I suppose the bottle authors have to add them?


Solution

  • bottle is not a statically-typed library (it offers no type annotations in bottle.py). Its organisation as a single bottle.py (instead of a package) also prevents you from making a py.typed yourself, which would otherwise allow mypy to pick up at least the class attributes, function/method signatures, and global variables (even if they don't have any typing information).

    The only way to get "real types" is for the bottle maintainers to add them. Assuming this isn't going to happen, you have several options:

    1. Generate a skeletal outline (Python stub file, extension .pyi) which will aid in basic code completion. This will catch variable, function, and class existence, but not variable and function signature typing information (simply because bottle.py doesn't have any in the first place).

      The skeletal outline can be generated in about a second using mypy's shipped stubgen tool, which you should already have if you've installed mypy.

    2. Collect runtime types by running pyannotate or MonkeyType on bottle's test suite, and use these tools to generate a .pyi. This is more accurate than just running stubgen, but the final quality is at the mercy of the test suite, and is very likely to result in a lot of false positive warnings by mypy, especially with descriptor classes like bottle's DictProperty and lazy_attribute.

    3. Use pytype's type inference to generate a .pyi. This generally results in more usable stubs, at the cost of an extremely long inference procedure that may end up building a dependency graph and scanning most of your site-packages, and will fail quite often if bottle has a lot of complex third-party dependencies. (EDIT: It looks like bottle only depends on the Python standard library, in which case you should get a fairly fast inference by pytype)

    You should end up getting a single bottle.pyi that you must move to the same directory as bottle.py. This should be in the site-packages of your virtual environment. Once you've done 2. or 3., mypy should be able to pick up typing errors properly.