pythonlxmlurllib3pyodide

imported module not found in class in Pyodide


I am trying to use pyodide with lxml and urllib3, for some reasons I don't understand, when I try to use urllib3 in a class supposed to be a Resolver for lxml etree I get an error NameError: name 'urllib3' is not defined.

Example code is online at https://martin-honnen.github.io/pyodide-tests/simple-requests-test-case3.html (note that all output goes to the browser console so use F12 to see the output).

Code is doing <script src="https://cdn.jsdelivr.net/pyodide/v0.26.0/full/pyodide.js"></script> and then


    const python = `
import js

import urllib3

import lxml

from lxml import etree as ET

url = base_url + 'foo-transform-module.xsl'

js.console.log(urllib3.request('GET', url).status)

class TestResolver(ET.Resolver):
    def resolve(self, url, id, context):
        print("Resolving URL '%s'" % url)
        if url.startswith('http'):
            return self.resolve_file(urllib3.request('GET', url), context)
        else:
            return False


parser = ET.XMLParser(no_network=False)
parser.resolvers.add(TestResolver())

tree = ET.parse(url, parser)

tree.getroot().tag
  `;

    async function main() {
        let pyodide = await loadPyodide();

        await pyodide.loadPackagesFromImports(python);

        const locals = new Map();
        locals.set('base_url', window.location.href.replace(/[^/]+?$/, ''));

        console.log(await pyodide.runPythonAsync(python, { locals : locals }));
    };

    main();

Full error in console is

Uncaught (in promise) PythonError: Traceback (most recent call last):
  File "/lib/python312.zip/_pyodide/_base.py", line 574, in eval_code_async
    await CodeRunner(
  File "/lib/python312.zip/_pyodide/_base.py", line 394, in run_async
    coroutine = eval(self.code, globals, locals)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<exec>", line 28, in <module>
  File "src/lxml/etree.pyx", line 3570, in lxml.etree.parse
  File "src/lxml/parser.pxi", line 1952, in lxml.etree._parseDocument
  File "src/lxml/parser.pxi", line 1978, in lxml.etree._parseDocumentFromURL
  File "src/lxml/parser.pxi", line 1881, in lxml.etree._parseDocFromFile
  File "src/lxml/parser.pxi", line 1200, in lxml.etree._BaseParser._parseDocFromFile
  File "src/lxml/parser.pxi", line 633, in lxml.etree._ParserContext._handleParseResultDoc
  File "src/lxml/parser.pxi", line 739, in lxml.etree._handleParseResult
  File "src/lxml/etree.pyx", line 329, in lxml.etree._ExceptionContext._raise_if_stored
  File "src/lxml/parser.pxi", line 462, in lxml.etree._local_resolver
  File "src/lxml/docloader.pxi", line 150, in lxml.etree._ResolverRegistry.resolve
  File "<exec>", line 20, in resolve
NameError: name 'urllib3' is not defined

    at new_error (pyodide.asm.js:10:9965)
    at pyodide.asm.wasm:0x16dbeb
    at pyodide.asm.wasm:0x177339
    at _PyEM_TrampolineCall_JS (pyodide.asm.js:10:125866)
    at pyodide.asm.wasm:0x1c2db7
    at pyodide.asm.wasm:0x2c7b17
    at pyodide.asm.wasm:0x20a78c
    at pyodide.asm.wasm:0x1c34a4
    at pyodide.asm.wasm:0x1c37b3
    at pyodide.asm.wasm:0x1c3831
    at pyodide.asm.wasm:0x29e865
    at pyodide.asm.wasm:0x2a4e5c
    at pyodide.asm.wasm:0x1c3971
    at pyodide.asm.wasm:0x1c35da
    at pyodide.asm.wasm:0x17699d
    at callPyObjectKwargs (pyodide.asm.js:10:64068)
    at Module.callPyObjectMaybePromising (pyodide.asm.js:10:65316)
    at wrapper (pyodide.asm.js:10:27006)
    at onGlobalMessage (pyodide.asm.js:10:101760)

Is there some flaw in my code? How do I get the imported module to be known in the code of the class?


Solution

  • I deleted old answer because it was wrong.


    It seems all problem makes locals - maybe it removes original urllib3

    I found Pyoide FAQ: How can I execute code in a custom namespace?.
    It shows some examples with globals.

    It seems this works for me:

    let my_namespace = pyodide.toPy({'base_url': window.location.href.replace(/[^/]+?$/, '')});
        
    await pyodide.runPythonAsync(python, {globals: my_namespace});
    

    Information for other users: problem was send by OP to Pyodide as issue

    imported module not found in class in pyodide, error "NameError: name 'urllib3' is not defined" · Issue #4834 · pyodide/pyodide

    but there is already similar issue

    Document weird behavior of runPython locals parameter · Issue #4673 · pyodide/pyodide