python-3.xazureazure-functionsponyorm

Azure Functions Python Vx - ModuleNotFound Exception with ORM


Context
I am using Azure Functions (I have tried both v1, and v2) to coordinate a set of scripts that trigger on HTTP. These scripts will pull data from an API, transform, then upload to a PostgreSQL database using the Pony ORM. The script works locally using Python 3.8, and the only reason I am tied to using Az Functions is for the below reasons:

  1. Functions natively support HTTP triggers for these scripts without any overhead manipulation on my end. This can be integrated into the downstream solution to allow the client to trigger a refresh of the data.

  2. Environment variables can be adjusted using the Azure portal, making it simpler for the non-tech clients to adjust the transformations without having to get into the system.

Problem
When I attempt to move my scripts into an Azure Function, I am unable run this locally, and am faced with a ModuleNotFound Exception with Pony. I have attempted to use other ORMs such as PeeWee, but run into the same issue.

My Attempted Solutions
I have verified that the requirements.txt file is up to date with all necessary packages. I have also manually installed these packages and verify that they appear in the venv folder. I have tried moving the database code into helper files (outside of the AzFunction) because for some reason I thought that might work, but to no avail. I will be attempting Django next, but am hesitant because I am unsure how it might affect performance (and it will require quite a refactor, it would be easier if I could just get Pony to work). From Microsoft Docs, I know that this may work better, as it relates to the linux-wheels and the respective packages having support for linux (as this is what Azure Functions run on). !

Output

When I run my functions locally, I receive this error:

 *  Executing task: .venv\Scripts\python -m pip install -r requirements.txt 

Requirement already satisfied: azure-functions in c:\users\...
Requirement already satisfied: pony in c:\{pwd}\.venv\lib\site-packages (from -r requirements.txt (line 6)) (0.7.17)
Requirement already satisfied: ratelimit in c:\{pwd}\.venv\lib\site-packages (from -r requirements.txt (line 7)) (2.2.1)
Requirement already satisfied: requests in c:\{pwd}\.venv\lib\site-packages (from -r requirements.txt (line 8)) (2.31.0)
Requirement already satisfied: charset-normalizer<4,>=2 in c:\{pwd}\.venv\lib\site-packages (from requests->-r requirements.txt (line 8)) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in c:\{pwd}\.venv\lib\site-packages (from requests->-r requirements.txt (line 8)) (3.4)
Requirement already satisfied: urllib3<3,>=1.21.1 in c:\{pwd}\.venv\lib\site-packages (from requests->-r requirements.txt (line 8)) (2.0.7)
Requirement already satisfied: certifi>=2017.4.17 in c:\{pwd}\.venv\lib\site-packages (from requests->-r requirements.txt (line 8)) (2023.7.22)

[notice] A new release of pip available: 22.3 -> 23.3.1
[notice] To update, run: C:\{pwd}\.venv\Scripts\python.exe -m pip install --upgrade pip
 *  Terminal will be reused by tasks, press any key to close it. 

 *  Executing task: .venv\Scripts\activate ; func host start 

Found Python version 3.8.10 (python3).

Azure Functions Core Tools
Core Tools Version:       4.0.5455 Commit hash: N/A  (64-bit)
Function Runtime Version: 4.27.5.21554

[2023-11-12T02:04:03.820Z] Customer packages not in sys path. This should never happen! 
[2023-11-12T02:04:06.309Z] Error in index_function_app. Sys Path: {$PATH}
[2023-11-12T02:04:06.334Z] Worker failed to index functions
[2023-11-12T02:04:06.344Z] Result: Failure
Exception: ModuleNotFoundError: No module named 'ratelimit'. Troubleshooting Guide: https://aka.ms/functions-modulenotfound
Stack:   File "C:\Users\Nathan Verghis\AppData\Roaming\npm\node_modules\azure-functions-core-tools\bin\workers\python\3.8/WINDOWS/X64\azure_functions_worker\dispatcher.py", line 345, in _handle__functions_metadata_request
    fx_metadata_results = self.index_functions(function_path)
  File "C:\Users\Nathan Verghis\AppData\Roaming\npm\node_modules\azure-functions-core-tools\bin\workers\python\3.8/WINDOWS/X64\azure_functions_worker\dispatcher.py", line 617, in index_functions
    indexed_functions = loader.index_function_app(function_path)
  File "C:\Users\Nathan Verghis\AppData\Roaming\npm\node_modules\azure-functions-core-tools\bin\workers\python\3.8/WINDOWS/X64\azure_functions_worker\utils\wrappers.py", line 48, in call
    raise extend_exception_message(e, message)
  File "C:\Users\Nathan Verghis\AppData\Roaming\npm\node_modules\azure-functions-core-tools\bin\workers\python\3.8/WINDOWS/X64\azure_functions_worker\utils\wrappers.py", line 44, in call
    return func(*args, **kwargs)
  File "C:\Users\Nathan Verghis\AppData\Roaming\npm\node_modules\azure-functions-core-tools\bin\workers\python\3.8/WINDOWS/X64\azure_functions_worker\loader.py", line 214, in index_function_app
    imported_module = importlib.import_module(module_name)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.2800.0_x64__qbz5n2kfra8p0\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "C:\{pwd}\function_app.py", line 4, in <module>
    from ratelimit import limits, sleep_and_retry
.
[2023-11-12T02:04:07.068Z] No job functions found. Try making your job classes and methods public. If you're using binding extensions (e.g. Azure Storage, ServiceBus, Timers, etc.) make sure you've called the registration method for the extension(s) in your startup code (e.g. builder.AddAzureStorage(), builder.AddServiceBus(), builder.AddTimers(), etc.).
For detailed output, run func with --verbose flag.

I will keep this updated if it works.If anyone knows a way to implement this functionality without Functions (while retaining the above benefits), or how to get the ORMs to correctly install on the Functions, do let me know. Thanks

-- UPDATE --

I was originally developing on Windows. Switching to Mac seemed to avoid the ModuleNotFoundError entirely. I discovered some bugs in the code I was deploying initially, and after fixing them was able to get the code working on Azure as well. Bringing the debugged code back to Windows and still no luck with local. At this point I'm just curious as to what caused the issue.


Solution

  • I tried the below Function code with pony-orm package connecting to PostgreSQL and it worked successfully like below:-

    Function Python - v1, Python version 3.10

    init.py code:-

    import os
    import json
    import azure.functions as func
    from pony.orm import Database, Required, db_session, select
    
    # Define your Pony ORM entities here
    db = Database()
    
    # Define your entity model (example)
    class Product(db.Entity):
        name = Required(str)
        price = Required(float)
    
    # Initialize Pony ORM database connection
    def initialize_database():
        db.bind(provider='postgres', user='siliconuser', password='xxxx',
                host='xxxxnpostgresql.postgres.database.azure.com', database='postgres')
        db.generate_mapping(create_tables=True)
    
    # Call the initialize function outside of a db_session
    initialize_database()
    
    # HTTP trigger function
    def main(req: func.HttpRequest) -> func.HttpResponse:
        with db_session:
            # Example: Inserting data into the database
            new_product = Product(name='Sample Product', price=20.0)
    
        with db_session:
            products = select(p for p in Product)[:]
            products_list = [{'name': p.name, 'price': p.price} for p in products]
    
        return func.HttpResponse(json.dumps(products_list), mimetype="application/json")
    

    My requirements.txt :-

    For pony package to work with postgresql psycopg2-binary package is also required.

    azure-functions
    
    pony
    
    psycopg2-binary
    

    Output:-

    While loading the Azure Functions locally and to install the package via virtual environment, Click on fn + f5 or Just f5 this will initialize the Function to run by itself along with venv package. Make sure you have Azure Functions Core tools installed in your system.

    enter image description here

    enter image description here

    And while deploying the Function instead of using, Azure Functions Extension, I have used Azure Functions Core tools command like below:-

    az login
    az account set --subscription "SID Subscription"
    func azure functionapp publish siliconfunc89
    

    Output:-

    enter image description here

    The HttpTrigger got deployed and running successfully refer below:-

    enter image description here

    Azure Function app Python version- 3.10 with Premium Function plan:-

    While importing large packages using Premium or Dedicated Function plan is more efficient.

    enter image description here

    Note- I am using Azure PostgreSQL.

    Also, In your error code- python Customer packages not in sys path. This should never happen! [2023-11-12T02:04:06.309Z] Error in index_function_app. Sys Path: {$PATH} [2023-11-12T02:04:06.334Z] Worker failed to index functions Error occurs when the python packages are not added in the correct folder while deployment. In order to resolve it and get more insights on the same refer this SO thread answer by Abdul Niyas P M and Pravallika. Make sure you downgrade your Azure Functions Core tools to python 4.0.534

    My Azure functions core tools version:-

    4.0.5148
    

    You can also initialize the virtual environment manually and then install package and run your function with func start command:-

    py -m venv .venv
    .venv\scripts\activate
    

    enter image description here