pythonaws-lambdapydanticaws-lambda-layerspy-langchain

AWS lambda "Unable to import module 'lambda_function': No module named >'pydantic_core._pydantic_core'" using langchain and lambda layers


Context

I have a lambda with the following handler function:

import json
from langchain_community.llms import ollama

def lambda_handler(event, context):
    return {
        'statusCode': 200,
        'body': json.dumps('Hello from LLM Lambda!')
    }

I have a .venv active and ran pip install langchain and then running pip list on my local I get:

aiohttp             3.9.1
aiosignal           1.3.1
annotated-types     0.6.0
anyio               4.2.0
async-timeout       4.0.3
attrs               23.2.0
certifi             2023.11.17
charset-normalizer  3.3.2
dataclasses-json    0.6.3
exceptiongroup      1.2.0
frozenlist          1.4.1
greenlet            3.0.3
idna                3.6
jsonpatch           1.33
jsonpointer         2.4
langchain           0.1.3
langchain-community 0.0.15
langchain-core      0.1.15
langsmith           0.0.83
marshmallow         3.20.2
multidict           6.0.4
mypy-extensions     1.0.0
numpy               1.26.3
packaging           23.2
pip                 23.3.2
pydantic            2.5.3
pydantic_core       2.14.6
PyYAML              6.0.1
requests            2.31.0
setuptools          58.0.4
sniffio             1.3.0
SQLAlchemy          2.0.25
tenacity            8.2.3
typing_extensions   4.9.0
typing-inspect      0.9.0
urllib3             2.1.0
yarl                1.9.4

As you can see the package pydantic_core is present.

From .venv/lib/python3.9/sites-packages I'm copiyng, renaming and zipping site-packages into python.zip then uploading it to an S3 bucket.

I then create a lambda layer with the following configs and referencing the mentioned zip file S3 object url like this https://bucketname.s3.us-east-2.amazonaws.com/lib/python.zip

enter image description here

THE PROBLEM

Is that the lambda function is currently crashing with the following error message:

{
  "errorMessage": "Unable to import module 'lambda_function': No module named 'pydantic_core._pydantic_core'",
  "errorType": "Runtime.ImportModuleError",
  "requestId": "7b2bba93-151f-4168-86fd-9ddad2d787fe",
  "stackTrace": []
}

Notes on what I have tried:


Solution

  • I found a solution to my own problem.

    Previously I was doing the pip install on my local environment, zipping the sites-packages and uploading it to S3

    My local environment is a Mac.

    I tested running the pip install inside an EC2 instance, zipping and uploading it straight to S3 and then the Lambda had no issues with any package.

    I'm guessing the difference in OS between my Mac and the Lambda was making the install of pydantic_core break at some point.