pythonpython-3.xpython-requestssubprocessmodulenotfounderror

Python script runs alone (from cmd) but not when it's in subprocess: error about requests module


I have a wrapper script that is running five other scripts in a subprocess and passing a config_file for each sub-script. First and last two scripts run without issues, but the middle one is causing issues. I have a virtual environment set up with all my dependencies, including the requests module. In the middle script I use the requests module to call an API.

This middle script works totally fine when I pass the config_file to it from the command prompt, but when I run it as part of the wrapper, it gives an error and says

Traceback (most recent call last):
  File "Leis.py", line 2, in <module>
    import requests
ModuleNotFoundError: No module named 'requests'

I have deactivated and removed my environment, re-set it up, confirmed I have requests in there. I can't seem to get inside the main function when running from wrapper, but it does fine on its own. Please help!

wrapper code:

for script in config["scripts"]:
        script_name = os.path.basename(script)
        logging.info(f"Running {script_name}")
        sp.run(["python", script, config_file], 
               stdout=sp.PIPE, stderr=sp.PIPE, 
               text=True, check=True, bufsize=1)
    logging.info(f".....")

leis.py code:

import requests
import sys
import os
import logging
from utils import setup_logging, load_config, check_files_exist
 

def get_registration(lei):
    try:
        url = f".........."
        payload = {}
        headers = {'Accept': 'application/vnd.api+json'}
        response = requests.request("GET", url, headers=headers, data=payload)
        json = response.json()
        if not json:
            return [None, None]
        registered_at_id = json.....

Solution

  • subprocess.run(["python", script…

    Are you sure python in the PATH is the right Python? When you want to run a script under the same interpreter as the current script use sys.executable:

    import sys, subprocess
    subprocess.run([sys.executable, script…
    

    On the other hand are you sure the current interpreter is the one that has access to the venv with requests?