laravelamazon-web-servicesaws-lambdabref

How to debug Bref Laravel Serverless Lambda Timeout?

I am using brief https://bref.sh/docs/frameworks/laravel to deploy in AWS Lambda my Laravel application (I build only one api without a frontend)

I am using in my composer.json the following versions

"laravel/framework": "10.22.0",
"bref/bref": "^2.1",
"bref/laravel-bridge": "^2.1",

My lambda dependencies are the following:

  1. one aws elasticcache instance located in the same vpc and private subnets
  2. one aws rds mysql instance located in the same vpc and private subnets. In addition this instance is publicly accessible for the experiment. I can connect to it with mysql client from my development machin as well
  3. a couple of public api endpoints reachable through the internet not related to me

My serverless.yml file looks like:

service: api

provider:
    name: aws
    region: eu-central-1
    logRetentionInDays: 7
    environment:
        LARAVEL_STORAGE_PATH: /tmp/storage
        # rest of env are loaded via the plugin and passed fine to the lambda config. No worries here

custom:
    serviceName: 'api' # the name of this service in AWS SSM store

package:
    # Files and directories to exclude from deployment
    patterns:
        - '!node_modules/**'
        - '!public/storage'
        - '!resources/assets/**'
        - '!storage/**'
        - '!tests/**'
        - '!.env'
        - '!build/**'
        - '!dev-tools/**'
        - '!ext-config/**'
        - '!queries/**'

functions:
    # This function runs the Laravel website/API
    web:
        handler: public/index.php
        vpc:
            securityGroupIds:
                - ${ssm:/${self:provider.stage}/${self:provider.region}/applications/${self:custom.serviceName}/lambda_security_group}
            subnetIds: ${ssm:/${self:provider.stage}/${self:provider.region}/infra/vpc_private_subnets}
        runtime: php-82-fpm
        timeout: 28 # in seconds (API Gateway has a timeout of 29 seconds)
        events:
            - httpApi: '*'

    # This function lets us run artisan commands in Lambda
    artisan:
        handler: artisan
        runtime: php-82-console
        timeout: 720 # in seconds
        events:
            - schedule:
                  rate: rate(1 minute)
                  input: '"schedule:run"'

plugins:
    - ./vendor/bref/bref                # lambda layer allowing laravel to work within aws serverless
    - serverless-lift                   # provides aws functionalities for creating the sqs queue
    - serverless-dotenv-plugin          # loads .env file into serverless variables

I can deploy totally fine with

php artisan config:clear
serverless deploy --stage development

After the lambda is available through AWS API Gateway I can

// 20230922174906
// https://6888rrxjik.execute-api.eu-central-1.amazonaws.com/api

{
  "message": "Internal Server Error"
}

What I troubleshooted so far

I looked into cloudwatch logs and do see that the lambda times-out

enter image description here

I studied brief docs related to connections to databases and reasons for timeout.

I could not see any reason of the failure because as mentioned I have same vpc_id and same subnets for both the database and redis, also lambda has same ones. In addition my database is publicly accessible.

I also checked my AWS configurations and I have one Internet gateway already connected to my VPC which is used in both redis,rds,lambda. I notice I don't have any NAT Gateway in my AWS account.

In addition I cannot see more detailed log in the stack trace to understand where this time is spent and what happens.

Any idea how to troubleshoot further and achieve the goal of having the lambda functional ?

Solution

  • UPDATE:

    I got it working by doing the following

    1. add bref extra extensions for redis
    - ./vendor/bref/extra-php-extensions        # for enabling php extensions via bref

            runtime: php-82-fpm
        layers:
            - ${bref-extra:redis-php-82}

    If I remove this I get again timeout. Probably because the redis related code is not available

    1. I also changed the lambda to use the public subnets instead of the private ones that I was using initially and shared in my question serverless.yml.
      • I am not sure if this was necessary because I can access the database and redis even if the lambda is on a private subnet (I tested this)
    2. Had done that already. Set LARAVEL_STORAGE_PATH=/tmp/storage as you cannot write elsewhere in the lambda.

    After those changes I can access the api that was timing out and I do confirm that

    1. connection to the database (which is in the public subnet as well) works (can fetch data)
    2. redis connection works. Created one endpoint to set cache key then scanned for it in redis

    I am testing if I can make api calls to other public apis/domains now when my lambda is in public subnets

    The conclusion is:

    1. A lambda inside a public subnet does not have internet access if the network gateway does not have a public ip assigned! (which by default it does not)

    So for the lambda to have internet access you need one of the following solutions:

    1. do not deploy it within a vpc, so no vpc attached
    2. deploy it inside a vpc public_subnet and attach an elastic ip to the newtork interface of the lambda (3 eip one for each network interface if you deploy to the availability zones a,b,c because you don't know where the lambda will run). You can find the network interfaces via the ui or cli/other automated tools
    3. You can deploy the lambda in a vpc private subnet and use NAT Gateway to provide internet access. Please note that this costs, nat gateway has around 35$/month cost and pay as you go for data. You can use nat instances which are cheaper but not so much reliable for huge traffic. Please note that you can use VPC endpoints for services like S3 and avoid getting charged for NAT traffic where it is not needed.

    PS: I am not sure if having the lambda in public subnet has any seccurity implication. The lambda itself has no api but is served through the aws api gateway