How do I debug a remote gunicorn Python web app using Jetbrains Gateway?

Background

I'm working as a programmer for a company that has a complicated setup of repos, so what they've done is set up an EC2 instance with all of the necessary repos and config, and I ssh into it to work on their Python (Flask) backend code. I've been using Jetbrains Gateway as my IDE (it runs an IDE instance on the remote server and then you connect to it with a local client interface, so it looks just like PyCharm but the actual code and IDE features are being executed on the remote server)

Problem

I can't figure out how to debug my code. I've used PyCharm's debug feature before but I'm feeling lost in this setup.

What I've tried so far

I looked at the "Edit Configurations" view and saw that there's a feature called "Python Debug Server", which requires that I specify the IP address of my local machine and a port it is listening on, which the remote EC2 instance will then make a TCP connection to. I don't have a static IP address and I'm not sure that my ISP allows incoming connections like that, so I was looking around online for a service that might help me with that, and saw that ngrok was mentioned in a few places.

I eventually got this working, and thought I'd write up how I did it for anyone else in the future who has this problem:

Solution

1. Set up an ngrok TCP tunnel.
   1. Sign up for ngrok's Personal plan ($10/mo).
      • How: Go to https://ngrok.com/pricing, click on "ngrok for development" and then "Get started" under the "Personal" plan.
        • 
      • Why: Jetbrains Gateway's Python Debug Server interface seems to require that you set a single port which (it seems) will be both the port that the remote IDE (running on an EC2 instance) will be reaching out to (trying to connect to your laptop) as well as the port that your local laptop client instance of Jetbrains Gateway will be listening for. But because ngrok's free tier both 1) requires that you set the port on your computer that should be reached out to by ngrok before it assigns the port ngrok will be listening on, as well as 2) randomly assigns the port it will be listening on externally, it seems you can't guarantee that those two ports used by ngrok will be the same unless you pay.
        • If anyone can get the TCP tunnel working on the free tier of ngrok, post a comment and I'll update this answer (or just update this answer yourself).
   2. In ngrok's web dashboard, click "TCP Addresses" and then "Add Address". Note the web address and port it assigns you.
      • 
   3. On your laptop, install the ngrok CLI.
   4. On your laptop, start the ngrok tunnel using the address and port from the dashboard.
      • Example: ngrok tcp --region=us --remote-addr=1.tcp.ngrok.io:12345 12345
2. Set up your app server on the remote server to be able to run the Python Debug Server module.
   1. On your laptop, go to the Edit Configurations dialog in Jetbrains Gateway.
      • 
   2. Create a new "Python Debug Server" entry.
      • 
      • Fill in the "IDE host name" and "Port" fields with the values you got from the ngrok web UI.
   3. On the remote server, install the pydevd module into the Python environment you normally use to run the server (whether that's a venv or a base interpreter).
      • Example: pip install pydevd-pycharm~=241.14494.200
   4. On the remote server, add the import pydevd_pycharm; pydevd_pycharm.settrace(...) code somewhere in the app server code where it'll be run (either when you start the app server or when you visit a route you want to debug).
3. Set up the Debug/Run configuration on your laptop.
   1. On the remote server, gather the following information:
      • The path to the Python executable you normally use to run the app server.
      • The path to the gunicorn executable you normally use to run the app server as well as any additional command-line options normally used when running it.
      • The working directory for the app server.
      • The paths to any .env files you normally use to run the app server.
      • Note: if you normally start your app server with a command like sudo systemctl start myappserver, you may find the values below in a file in your /etc/systemd/system/ directory. So you may want to run something like cat /etc/systemd/system/myappserver.service.
        • 
   2. Create a new Python Run/Debug configuration.
   3. Copy in the values you gathered earlier.
      • 
      • You may need to add the Python interpreter by going to File-->Settings-->Project: app_name-->Python Interpreter before it will show up as an option in this dialog.
4. Run the app server through the debugger.
   1. If you have the app server already running normally, stop it. You may need to run a command like sudo systemctl stop myappserver.
   2. Click the green debug button in Jetbrains to start the app server with the debugger active.
      • 
5. The app should now start up and pause at the point where you included the settrace() command. Any breakpoints you add to the code should now work (execution should stop at the breakpoint).

Additional resources / troubleshooting help

• This was the best resource I came across: Jetbrains - Remote Debugging with PyCharm.
• If you have trouble with the steps above, I recommend the following (which I also did):
  1. Try running a "Hello world"-type Python file on the remote server in debug mode while using the remote interpreter method of debugging, explained here.
     • This avoids the complication of needing to get the TCP tunnel working.
  2. Try running a "Hello world"-type Python file on the remote server while using the "Python Debug Server" method described above.
     • This avoids the additional complication of needing to run the code through gunicorn and have the web app configuration correct.
     •