pythonnode.jspython-importchild-processspawn

NodeJS, AWS Amplify, Python PythonShell local imports module not found


Apologies in advance for the text wall, it's a little hard to explain this one.

So I have an application, that was not designed by myself but I am now in charge of it. I am extremely new to JavaScript/CSS, nodeJS, and Python, but have a pretty decent understanding of AWS and programming in general. Just the syntax and other language specific stuff that gets me.

The issue is that when using the PythonShell or child-process.spawn method of running Python code with nodeJS I receive a module not found error when trying to import locally in the python code.

Here is the javaScript code I am running:

const {PythonShell} = require('python-shell');
let pyshell = new PythonShell('../backend/src/functions/resource_logs.py');

// sends a message to the Python script via stdin
// pyshell.send('hello');

pyshell.on('message', function (message) {
// received a message sent from the Python script (a simple "print" statement)
console.log(message);
});

// end the input stream and allow the process to exit
pyshell.end(function (err,code,signal) {
if (err) throw err;
console.log('The exit code was: ' + code);
console.log('The exit signal was: ' + signal);
console.log('finished');
});

And then this is the Python code I am running at resource_logs.py that causes no error and outputs the desired printed line:

import sys
import os

print("Printed in the python script") #Print line to test pythonShell

VS the code I use when attempting to locally import:

import sys
import os

from utils import secrets
print("Printed in the python script") #Print line to test pythonShell

Which then throws the following error message:

if (err) throw err;
            ^

PythonShellError: ModuleNotFoundError: No module named 'utils' at PythonShell.parseError (C:\Programming\TIW\fabmansync\node_modules\python-shell\index.js:303:21) at terminateIfNeeded (C:\Programming\TIW\fabmansync\node_modules\python-shell\index.js:193:32) at ChildProcess.<anonymous> (C:\Programming\TIW\fabmansync\node_modules\python-shell\index.js:185:13) at ChildProcess.emit (node:events:519:28) at ChildProcess._handle.onexit (node:internal/child_process:294:12) Python Traceback ----- File "C:\Programming\TIW\fabmansync\backend\src\functions\resource_logs.py", line 13, in <module> from utils import secrets { traceback: 'Traceback (most recent call last):\r\n' + File "C:\\Programming\\TIW\\fabmansync\\backend\\src\\functions\\resource_logs.py", line 13, in <module>\r\n' + from utils import secrets\r\n' +
    "ModuleNotFoundError: No module named 'utils'\r\n", executable: 'python', options: null, script: '..\\backend\\src\\functions\\resource_logs.py', args: null, exitCode: 1 }

Be sure to note it isn't just all imports as os and sys are fine just the local one. I also just noticed that it's saying utils\r\n if anyone could explain why that is?

A brief idea of the file structure (all places that should have an init.py have it): fabmansync has src and backend. Src then has the JS file. Meanwhile backend has it's own src file, which has files utils and functions, and secrets.py is in utils and resoruce_logs.py is in functions.

I know for a fact that these imports can and do work when not being run by the JS file so it has something to do with Pythonshell/child-process.spawn not liking these imports without a doubt.

I am not looking for a work around or some other solution to my issue. I have another idea that's tedious so if nothing can come of this I can do that. And I'm also not going to be using express/ the app function for any reason as the nodeJS AWS hookup can automatically change the webpage I'm using.

Please feel free to ask any questions for additional info, and thank you for any assistance you can provide on solving this problem I've been trying for like two days now.

I've done a ton of googling and brief skimming of documentation. So rundown of what I've tried, that I can remember:

  1. child-process.spawn, I've tried using this extensively and only actually switched to pythonshell recently cause I thought it would be more helpful, but I guess it just uses spawn under the hood.

  2. CWD: with child-process.spawn, this just never really went anywhere might have not used it correctly.

  3. Moving functions and utils into fabmansync/src. This one did work in that it allowed me to import secrets, however secrets has imports such as boto3.session and botocore.exceptions that would throw the error just the same. I could try moving these as well but then we've got a rinse and repeat of moving a million import files somewhere else in the project when that shouldn't really be necessary.

Expected output: The python script should find the module. Same as it does when it's not run by the javaScript file, and then those imports should successfully import their imports etc. And then print the line.


Solution

  • Time to tackle that utils library install.

    pip install utils