Converting a Codebase to mypy

31 Oct 2019

mypy is a static type checker for Python. I’ve tried twice before to use it on an existing codebase but was always turned off by all of the initial import errors. I’ve tried again and have succeeded in getting past the initial pain. In this post I will document my journey.

The project I’m adding mypy to is called ‘learn’. It is a webapp using flask. It has 3.5k lines of Python; so not too large.

$ mypy learn
learn/strings.py:7: error: Cannot find module named 'flask_babel'
learn/__init__.py:13: error: Cannot find module named 'flask_sqlalchemy'
learn/__init__.py:13: note: See https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports
learn/__init__.py:14: error: Cannot find module named 'flask_login'
learn/__init__.py:15: error: Cannot find module named 'flask_babel'
learn/logs.py:13: error: Cannot find module named 'flask_login'
learn/logs.py:17: error: Item "None" of "Optional[Any]" has no attribute "before_request"
learn/logs.py:28: error: Item "None" of "Optional[Any]" has no attribute "teardown_request"
learn/errors.py:9: error: Item "None" of "Optional[Any]" has no attribute "errorhandler"
learn/errors.py:14: error: Item "None" of "Optional[Any]" has no attribute "errorhandler"
learn/models/user.py:7: error: Cannot find module named 'flask_login'
learn/models/user.py:12: error: Name 'db.Model' is not defined
learn/models/user.py:15: error: Item "None" of "Optional[Any]" has no attribute "Column"
learn/models/user.py:15: error: Item "None" of "Optional[Any]" has no attribute "Integer"
learn/models/user.py:16: error: Item "None" of "Optional[Any]" has no attribute "Column"

[many more lines]

Found 371 errors in 15 files (checked 17 source files)

Lots of errors.

I remembered that some libraries probably don’t have stub files. So I created stubs for flask_babel, flask_sqlalchemy, and flask_login.

$ touch stubs/flask_babel.pyi
$ touch stubs/flask_sqlalchemy.pyi
$ touch stubs/flask_login.pyi
$ MYPYPATH=./stubs mypy learn

Less errors now. I next chose one error and tried to fix it.

learn/routes/question.py:11: error: Module 'flask_login' has no attribute 'login_required'

The line it points to is

from flask_login import login_required

So it seems I first need to add stuff to the stubs. Specifically, everything I import from any library without its own types I’ll have to add to the stub file and give it a type (I could just give it the type Any, but that won’t do much good)

So, login_required is a decorator. I found some info in a mypy issue on typing decorators.

Since login_required won’t change the type of the function it decorates, I choose to type it as follows:

from typing import TypeVar, Callable

T = TypeVar('T')

def login_required(f: T) -> T: ...

I’ll now choose a different error.

learn/routes/survey.py:10: error: Module 'flask_login' has no attribute 'current_user'

I want to give this the type learn/models/user/User from my own code. To do this I imported the class in the stub file.

from learn.models import User

From there I kept adding types to my code. I quickly realized that the body of function won’t be typechecked unless the function itself is given a type. This can be changed by using the --strict flag for mypy. However that becomes a little too strict initially.

Currently I’m going through the codebase enabling, one at a time, the flags which --strict enables. I’ve found if I enable them all at once then there’s too many errors to focus on.

Practically, I haven’t found any huge bugs yet, but mypy has caught a few instances where, in a list comprehension, I do operations without checking for None when dealing with Optional[Blah].

The big win so far is with documentation. There are a lot of functions that I have forgotten exactly what they return. Using mypy has forced me to figure it out and then add the type, which is a form of checkable documentation.

It’s also improved my code. For example, there were times which I would reassign a variable to some modified version of itself, taking on a new type. This code is hard to maintain. So when mypy shows me exactly where I’m doing this I have an opportunity to refactor it.

View on GitHub