When browsing through my website or when I refresh page, it happens that Django forgets about the authentication state and returns the page that an anonymous user would see. When the page is refreshed again it either returns to a logged in session or still shows the unauthenticated state. It's random.
I checked the request headers that are sent by the browser and sessionid is there on every request. Even when the cycle changes from authenticated → unauthenticated → authenticated the sessionid stays the same. So the session is not deleted or purged from the session backend because the session id can still be used after seeing a page in anonymous state. The session is just not picked up by Django.
The website is hosted on render.com, a platform-as-a-service just like Heroku or fly.io. In the application logs I noticed a message Session data corrupted. The metrics showed that autoscaling kicked in and there had been three instances of the application.
This means that one of the three instances was able to return a page in authenticated state belonging to the session cookie while the other two instances were not and instead returned Session data corrupted.
Django uses settings.SECRET_KEY for signing session cookies. Unfortunately I had made the mistake to set a random secret key:
SECRET_KEY = os.getenv("DJANGO_SECRET_KEY", get_random_secret_key())
This meant not only that on every deploy, a new SECRET_KEY would be generated, killing all existing sessions. But also that instances are not able to use the session another instance created because the signing wouldn't match and thus users would cycle between authenticated and unauthenticated state.
Defining SECRET_KEY in the environment is the solution to this problem.