What I want to achieve

To scrape an website using AWS Lambda and save the data on S3.

The issues I'm having

When I execute Lambda, the following error message appears.

{ "errorMessage": "Unable to import module 'lambda_function': cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_' (/opt/python/urllib3/util/ssl_.py)", "errorType": "Runtime.ImportModuleError", "requestId": "fb66bea9-cbad-4bd3-bd4d-6125454e21be", "stackTrace": [] }

Code

The minimum Lambda code is as follows.

import requests
import boto3 

def lambda_handler(event, context):
    s3 = boto3.client('s3')
    upload_res = s3.put_object(Bucket='horserace-dx', Key='/raw/a.html', Body='testtext')
    
    return event

An layer was added to the Lambda. Files were save in python folder using the commands below , frozen in a zip file, then uploaded to AWS Lambda as a layer.

!pip install requests -t ./python --no-user
!pip install pandas -t ./python --no-user
!pip install beautifulsoup4 -t ./python --no-user

• The bucket horserace-dx exists
• The folder raw exists
• The role of the Lambda is properly set. It can read from and write to S3
• The runtime of the Lambda is Python 3.9. The python version of the local computer is 3.9.13.

What I did so far

I google "cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_'" and found some suggestions. I made the layer with the following code and tried again in vain.

!pip install requests -t ./python --no-user
!pip install pandas -t ./python --no-user
!pip install beautifulsoup4 -t ./python --no-user
!pip install urllib3==1.26.15 -t ./python --no-user

So what should I do to achieve what I want to achieve? Any suggestions would be greatly appreciated.

1. Execute the following commands.

   pip install requests==2.25.0 -t ./python --no-user pip install beautifulsoup4 -t ./python --no-user pip install pytz -t ./python --no-user

2. On PyPI, download the following whl files from the pages of numpy and pandas

• numpy-1.24.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
• pandas-2.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

3. Unzip the files and move the contents to the python folder.

4. Zip the python folder and upload it to AWS Lambda Layer.

5. Set the layer to the Lambda.

6. Then the code runs without errors.