I have a mangum python service with a cdk stack that I can deploy on my local box just fine. Here are my:
pyproject.toml
[tool.poetry]
name = "dics-core-data-service"
version = "0.1.0"
description = ""
authors = ["Clayton <clayton.stetz@gmail.com>"]
readme = "README.md"
package-mode = false
[tool.poetry.dependencies]
python = "^3.12"
fastapi = "^0.115.0"
uvicorn = "^0.31.0"
python-dotenv = "^1.0.1"
pydantic = {extras = ["email"], version = "^2.9.2"}
phonenumbers = "^8.13.46"
pydantic-extra-types = "^2.9.0"
mangum = "^0.19.0"
constructs = "^10.3.0"
psycopg2-binary = "^2.9.9"
[tool.poetry.dev-dependencies]
pytest = "^8.3.3"
aws-cdk-lib = "^2.161.1"
aws-cdk-aws-lambda-python-alpha = "^2.161.1a0"
[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
relevant sections of poetry.lock
[[package]]
name = "aws-cdk-lib"
version = "2.161.1"
description = "Version 2 of the AWS Cloud Development Kit library"
optional = false
python-versions = "~=3.8"
files = [
{file = "aws_cdk_lib-2.161.1-py3-none-any.whl", hash = "sha256:c7de930396b1b9f0f512a728a1b926c77c3cab28fbc11fd4f81819dd9563bfb3"},
{file = "aws_cdk_lib-2.161.1.tar.gz", hash = "sha256:e27a427bc6d95dd2eb0500b0de628a88ee587b212f999fa6efb7e9ab17980201"},
]
[package.dependencies]
"aws-cdk.asset-awscli-v1" = ">=2.2.202,<3.0.0"
"aws-cdk.asset-kubectl-v20" = ">=2.1.2,<3.0.0"
"aws-cdk.asset-node-proxy-agent-v6" = ">=2.1.0,<3.0.0"
"aws-cdk.cloud-assembly-schema" = ">=38.0.0,<39.0.0"
constructs = ">=10.0.0,<11.0.0"
jsii = ">=1.103.1,<2.0.0"
publication = ">=0.0.3"
typeguard = ">=2.13.3,<5.0.0"
[[package]]
name = "aws-cdk-aws-lambda-python-alpha"
version = "2.161.1a0"
description = "The CDK Construct Library for AWS Lambda in Python"
optional = false
python-versions = "~=3.8"
files = [
{file = "aws_cdk.aws_lambda_python_alpha-2.161.1a0-py3-none-any.whl", hash = "sha256:bc50b108080d06c68d0d8468467b59751082e1a7b553452a92175bf03e38d0aa"},
{file = "aws_cdk_aws_lambda_python_alpha-2.161.1a0.tar.gz", hash = "sha256:3543cbaeabb6fb2c8e694cf5c2525ab5f1965130cc7db79a929c8c38be40db84"},
]
[package.dependencies]
aws-cdk-lib = ">=2.161.1,<3.0.0"
constructs = ">=10.0.0,<11.0.0"
jsii = ">=1.103.1,<2.0.0"
publication = ">=0.0.3"
typeguard = ">=2.13.3,<5.0.0"
If I run poetry install on my local box then run cdk deploy --context stage="dev", it succeeds. It sees my dev dependencies just fine and can run the cdk code I have that requires aws-cdk-lib.
However, I need to run this in github actions as well so that I can deploy changesets when the "dev" and "stage" and "prod" branches get pushed to. This is where I hit issues.
Here's my github actions yaml:
name: AWS Service CI/CD
on:
push:
branches: [dev]
jobs:
build:
environment: dev
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Python 3.12
uses: actions/setup-python@v4
with:
python-version: "3.12"
- name: Set up Node
uses: actions/setup-node@v3
with:
node-version: "22"
- name: Service - Install Python and CDK
run: |
cd serverlessservice
python -m pip install --upgrade pip
npm install -g aws-cdk
- name: Service - Install Poetry
uses: Gr1N/setup-poetry@v8
- name: Service - Configure AWS credentials
uses: aws-actions/configure-aws-credentials@master
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: "us-west-2"
- name: Service - Install Dependencies
run: |
cd serverlessservice
poetry install
cdk deploy --context stage="dev"
The very last line fails and the output is as follows:
Traceback (most recent call last):
File "/home/runner/work/core-data-service/core-data-service/serverlessservice/./infra.py", line 2, in <module>
import aws_cdk as cdk
ModuleNotFoundError: No module named 'aws_cdk'
Subprocess exited with error 1
If I manually install aws-cdk-lib, the error changes to complain about the other dev dependency I have listed, which is my main evidence that it's the dev dependencies that are the issue.
If I manually install all my dev dependencies needed by my cdk-deploy call using pip install, it DOES work but then the deploy package is way too big and gets rejected, because aws-cdk-lib is huge and should not be in production deploys.
This is the final blocker in a long line of issues with poetry and github actions, someone save me :D
TL;DR: run poetry run cdk deploy --context stage="dev" instead of cdk deploy --context stage="dev"
How do you do the deployment locally? Because right now you are not activating your poetry-generated virtual environment within the GH actions.
The problem should be in this part:
run: |
cd serverlessservice
poetry install
cdk deploy --context stage="dev"
You do poetry install, which installs your dependencies, including aws_cdk, but you never activate the virtual environment created by poetry. Your next step fails because it does not use the libraries installed in the previous step, but the libraries available globally for the build machine.
So do this instead:
run: |
cd serverlessservice
poetry install
poetry run cdk deploy --context stage="dev"
poetry run does two things at the same time: first it activates the virtual environment that contains your libraries and then it runs the command given to it with the environment active.
You can verify this simply by running something like this:
run: |
cd serverlessservice
poetry install
which python
poetry run which python
This should give you something like this as an output:
/opt/hostedtoolcache/Python/3.12.7/x64/bin/python
/home/runner/.cache/pypoetry/virtualenvs/serverlessservice-jiIPjY84-py3.12/bin/python