Module not found if graalpy is packaged in Spring Boot Jar

I'm building a Spring Boot (3 RC1) application with some Python code (GraalVM 22.3). Running the application in dev mode works. After building Jar with Maven I get an error:


Caused by: org.graalvm.polyglot.PolyglotException: ModuleNotFoundError: No module named 'pystac'
    at org.graalvm.sdk/org.graalvm.polyglot.Context.eval(Context.java:399) ~[org.graalvm.sdk:na]
    at ch.so.agi.sodata.stac.ConfigService.readXml(ConfigService.java:116) ~[classes!/:0.0.1-SNAPSHOT]
    at ch.so.agi.sodata.stac.SodataStacApplication.lambda$init$0(SodataStacApplication.java:60) ~[classes!/:0.0.1-SNAPSHOT]
    at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:767) ~[spring-boot-3.0.0-RC1.jar!/:3.0.0-RC1]
    ... 13 common frames omitted

The python.Executable shows to the graalpy executable packaged in the Jar: file:/Users/stefan/sources/datenbezug/sodata-stac/target/sodata-stac-0.0.1-SNAPSHOT.jar!/BOOT-INF/classes!/venv/bin/graalpy

Grepping the Jar shows the pystac module in the Jar file, e.g. BOOT-INF/classes/venv/lib/python3.8/site-packages/pystac/item.py

Creating the context with:

var VENV_EXECUTABLE = ConfigService.class.getClassLoader()
                .getResource(Paths.get("venv", "bin", "graalpy").toString())
                .getPath();

var context = Context.newBuilder("python")
                .allowAllAccess(true)
                .option("python.Executable", VENV_EXECUTABLE)
                .option("python.ForceImportSite", "true")
                .build()

Is it possible to put the whole python stuff including third party libs into the FatJar? Or did I just miss some packaging magic?

Solution

The issue is that the default Truffle filesystem, AFAIK, only supports the actual filesystem of your OS, i.e., it does not "see" resources in the jar file. This is why it works in dev mode when the resources are just files on the filesystem.

There are two options how to deal with this:

do not put the venv into resources, but deploy it separately, such that it ends up somewhere on the filesystem
implement custom Truffle filesystem, but I am not aware of anyone who tried this with virtual environments yet, so there may be some rough edges. Most notably GraalVM LLVM Toolchain that is used to build Python native extensions for execution with GraalVM LLVM engine produces system specific binaries, so your jar will not be as portable as one would expect from a jar. If you only use pure Python packages, it may be fine.

Update as of GraalPy 24.0.0+:

GraalPy now provides a Maven archetype to generate a skeleton of Maven based Java project that puts the venv into resources and implements custom Truffle filesystem that accesses the resources.

mvn archetype:generate \
  -DarchetypeGroupId=org.graalvm.python \
  -DarchetypeArtifactId=graalpy-archetype-polyglot-app \
  -DarchetypeVersion=24.0.0

There is also GraalPy Maven plugin that simplifies Python packages handling - you can specify the Python packages in the pom.xml and the virtual environment is created and populated transparently as part of mvn package. This plugin is used in the project generated by the archetype.

See https://www.graalvm.org/latest/reference-manual/python for more details.