javapythongraalvmgraalpython

run Python in GraalVM - `import requests` fails with Exception: You need either charset_normalizer or chardet installed (requests init failure)


I try to run Python script in GraalVm and it fails on

import requests

And it seems GraalVM issue, as the script work with usual python3.

Java code:

import org.graalvm.polyglot.Context;
import org.graalvm.polyglot.Source;

import java.io.BufferedReader;
import java.io.FileReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.nio.file.Paths;

public class RunPython4 {
    public static String PYTHON = "python";
    // venv/**/* must be included into resources in pom.xml
    private static String VENV_EXECUTABLE = RunPython4.class.getClassLoader().getResource(Paths.get("venv", "bin", "graalpython").toString()).getPath();
    private static String SOURCE_FILE_NAME = "health.py";

    public static void log(String s){
        System.out.println(s);
    }

    public static void main(String[] args) {
        log("Hello Java!");
        log(System.getProperty("java.version"));
        log(System.getProperty("java.runtime.version"));

        String pyFilename = "./health.py";

        try (Context context = Context.newBuilder("python").
                allowAllAccess(true).
                option("python.ForceImportSite", "true").
                option("python.Executable", VENV_EXECUTABLE).
                build();) {
            context.eval(PYTHON, "print('Hello Python!')");

            context.eval(PYTHON, "import sys; print(sys.version)");

            //4
            try(BufferedReader br = new BufferedReader(new FileReader(pyFilename))) {
                int i = 0;
                for(String line; (line = br.readLine()) != null; ) {
                    i++;
                    log(""+i+": "+line);
                    context.eval("python", line);
                }
            }catch ( IOException e){
                log("IOException "+e);
            }

Output:

Hello Java!
11.0.11
11.0.11+8-jvmci-21.1-b05
Hello Python!
3.8.5 (Fri Jun 25 17:55:09 CST 2021)
[Graal, GraalVM CE, Java 11.0.11]
1: # Checking subgraph health v1.1
2: # https://thegraph.com/docs/deploy-a-subgraph#checking-subgraph-health
3: 
4: import requests
Exception in thread "main" Exception: You need either charset_normalizer or chardet installed
    at org.graalvm.sdk/org.graalvm.polyglot.Context.eval(Context.java:379)
    at RunPython4.main(RunPython4.java:54)

Process finished with exit code 1

I searched deeper, "You need either charset_normalizer or chardet installed" is string from

https://github.com/psf/requests/blob/master/requests/__init__.py#L57-L85

def check_compatibility(urllib3_version, chardet_version, charset_normalizer_version):
    urllib3_version = urllib3_version.split('.')
    assert urllib3_version != ['dev']  # Verify urllib3 isn't installed from git.

    # Sometimes, urllib3 only reports its version as 16.1.
    if len(urllib3_version) == 2:
        urllib3_version.append('0')

    # Check urllib3 for compatibility.
    major, minor, patch = urllib3_version  # noqa: F811
    major, minor, patch = int(major), int(minor), int(patch)
    # urllib3 >= 1.21.1, <= 1.26
    assert major == 1
    assert minor >= 21
    assert minor <= 26

    # Check charset_normalizer for compatibility.
    if chardet_version:
        major, minor, patch = chardet_version.split('.')[:3]
        major, minor, patch = int(major), int(minor), int(patch)
        # chardet_version >= 3.0.2, < 5.0.0
        assert (3, 0, 2) <= (major, minor, patch) < (5, 0, 0)
    elif charset_normalizer_version:
        major, minor, patch = charset_normalizer_version.split('.')[:3]
        major, minor, patch = int(major), int(minor), int(patch)
        # charset_normalizer >= 2.0.0 < 3.0.0
        assert (2, 0, 0) <= (major, minor, patch) < (3, 0, 0)
    else:
        raise Exception("You need either charset_normalizer or chardet installed")

So it is clear that requests init fails over charset_normalizer_version that is likely undefined

https://github.com/psf/requests/blob/master/requests/__init__.py#L47

try:
    from charset_normalizer import __version__ as charset_normalizer_version
except ImportError:
    charset_normalizer_version = None

The requests package is present of course

enter image description here


Solution

  • The problem was, that I was installing requests using system python and not with graalpython -m ginstall install requests

    Some packages are complex to compile, and to be worry free use prepared (precompiled?/fixed) packages.

    See In graalpython what is difference between `ginstall` and `pip`?

    Runnable code is at https://github.com/paulvi/graalpython-java-template