python-packagingpython-importlibpantspython-pex

Importing json resources inside .pex (Python Executable (format by Twitter))


I'm using a Twitter engineered build tool pants to manage many projects inside my monorepo. It outputs .pex files when I complete a build, this is a binary that packages the bare minimum dependencies I need for each project and makes them a "binary" (actually an archive that's decompressed at runtime), my issue is a utility that my code has used for a long time fails to detect some .json files(now that I'm using pants) I have stored under my environments library. all my other code seems to run fine. I'm pretty sure it has to do with my config, perhaps I'm not storing the resources properly so my code can find it, though when I use unzip my_app.pex the resources I desire are in the package and located in the proper location(dir). Here is the method my utility uses to load the json resources:

if test_env:
    file_name = "test_env.json"
elif os.environ["ENVIRONMENT_TYPE"] == "PROD":
    file_name = "prod_env.json"
else:
    file_name = "dev_env.json"
try:
    json_file = importlib.resources.read_text("my_apps.environments", file_name)
except FileNotFoundError:
    logger.error(f"my_apps.environments->{file_name} was not found")
    exit()
config = json.loads(json_file)

here is the the BUILD file I use for these resource currently:

python_library(
   dependencies=[
       ":dev_env",
       ":prod_env",
       ":test_env"
   ]
 )

resources(
   name="dev_env",
   sources=["dev_env.json"]
)

resources(
   name="prod_env",
   sources=["prod_env.json"]
)

resources(
   name="test_env",
   sources=["test_env.json"]
)

and here is the BUILD file for the utility that calls these resources of which the python code above is what you saw:

python_library(
   name="environment_handler",
   sources=["environment_handler.py"],
   dependencies=[
       "my_apps/environments:dev_env",
       "my_apps/environments:prod_env",
       "my_apps/environments:test_env"
   ]
)

I always get an FileNotFoundError exception and I'm confused because the files are available to the runtime, what's causing these files to not be accessible? and is there a different format I need to set up the JSON resources as?

Also for context here is the decompressed .pex file(actually just the source-code dir):

    ├── apps
│   ├── __init__.py
│   └── services
│       ├── charts
│       │   ├── crud
│       │   │   ├── __init__.py
│       │   │   └── patch.py
│       │   ├── __init__.py
│       │   └── main.py
│       └── __init__.py
├── environments
│   ├── dev_env.json
│   ├── prod_env.json
│   └── test_env.json
├── __init__.py
├── models
│   ├── charts
│   │   ├── base.py
│   │   └── __init__.py
│   └── __init__.py
└── utils
    ├── api_queries
    │   ├── common
    │   │   ├── connections.py
    │   │   └── __init__.py
    │   └── __init__.py
    ├── calculations
    │   ├── common
    │   │   ├── __init__.py
    │   │   └── merged_user_management.py
    │   └── __init__.py
    ├── environment_handler.py
    ├── __init__.py
    ├── json_response_toolset.py
    └── security_toolset.py

Solution

  • I figured it out: I changed the way I access the files within the library and it works perfectly before and after the build to .pex format. I used:

    import pkgutil
    #json_file = importlib.resources.read_text("my_apps.environments", file_name)
    json_file = pkgutil.get_data("my_apps.environments", file_name).decode("utf-8")