Firestore SDK hangs in production

I'm using the Firebase Admin Python SDK to read/write data to Firestore. I've created a service account with the necessary permissions and saved the credentials .json file in the source code (I know this isn't the most secure, but I want to get the thing running before fixing security issues). When testing the integration locally, it works flawlessly. But after deploying to GCP, where our service is hosted, calls to Firestore don't work properly and retry for a while before throwing 503 Deadline Exceeded errors. However, SSHing into a GKE pod and calling the SDK manually works without issues. It's just when the SDK is used in code flow that causes problems.

Our service runs in Google Kubernetes Engine in one project (call it Project A), but the Firestore database is in another project (call it project B). The service account that I'm trying to use is owned by Project B, so it should still be able to access the database even when it is being initialized from inside Project A.

Here's how I'm initiating the SDK:

from firebase_admin import get_app
from firebase_admin import initialize_app
from firebase_admin.credentials import Certificate
from firebase_admin.firestore import client
from google.api_core.exceptions import AlreadyExists

credentials = Certificate("/path/to/credentials.json")
try:
    app = initialize_app(credential=credentials, name="app_name")
except ValueError:
    app = get_app(name="app_name")
client = client(app=app)

Another wrinkle is that another part of our code is able to successfully use the same service account to produce Firebase Access Tokens. The successful code is:

import firebase_admin
from firebase_admin import auth as firebase_admin_auth

if "app_name" in firebase_admin._apps:
    # Already initialized
    app = firebase_admin.get_app(name="app_name")
else:
    # Initialize
    credentials = firebase_admin.credentials.Certificate("/path/to/credentials.json")
    app = firebase_admin.initialize_app(credential=credentials, name="app_name")

firebase_token = firebase_admin_auth.create_custom_token(
    uid="id-of-user",
    developer_claims={"admin": is_admin, "site_slugs": read_write_site_slugs},
    app=app,
)

Any help appreciated.

Turns out that the problem here was a conflict between gunicorn's gevents and the SDK's use of gRCP. Something related to websockets. I found the solution here. I added the following code to our Django app's settings:

import grpc.experimental.gevent as grpc_gevent
grpc_gevent.init_gevent()

EDIT

It's important that this patch is only applied in environments where gunicorn is running, or you can cause problems instead of fixing them. I ran into this problem because the same code powers deployments that handle asynchronous tasks, which don't use gunicorn. I recommend the following conditional statement to ensure that this patch is only called when gevent is in use:

import socket
import gevent.socket

# If Gunicorn has run gevent.monkey.patch_all, patch gevent to be compatible with gRPC
if socket.socket is gevent.socket.socket:
    import grpc.experimental.gevent as grpc_gevent
    grpc_gevent.init_gevent()