[SOLVED] Python 3.6 unavailable in AWS CodeBuild, Python 3.5 unavailable in AWS Lambda

Python 3.6 unavailable in AWS CodeBuild, Python 3.5 unavailable in AWS Lambda

I have a Python 3 project which I am trying to deploy to AWS Lambda via AWS Codestar -> Codepipeline -> Codebuild -> Cloudformation.

My project (which really just consists of a simple API Gateway handler method) imports a Python 3 (requires 3) project (newspaper). I am using Virtualenv 15.1.0 on my home computer and if I install Newspaper with Python 3.5 and then upload to Lambda (Python 3.6 runtime), it throws errors related to PIL / Pillow.

First it says it can't find _image, which appears to be resolved by deleting the PIL directory in site-packages, however that just results in it throwing the error that it can't find PIL.

If, however, I build with Python 3.6 and then upload to Lambda, it works just fine (whether I delete PIL or not).

So, that appears to me that I can't install Newspaper with 3.5 and try to execute in a 3.6 runtime.

So, now I am trying to deploy via Codestar, however Codestar seems to default to aws/codebuild/eb-nodejs-4.4.6-amazonlinux-64:2.1.3, even for Python projects, and all it seems to have available in the Yum repository is Python 3.5 and of course Lambda only has the 3.6 runtime.

Even if I switch the image within Codebuild itself, there don't seem to be any images built with the Python3.6 runtime (according to the documentation). Even the Docker images seem to lack Python 3.6.

So, I am trying to install Python 3.6 in Codebuild during the INSTALL phase in my buildspec.yml file, however I can't find the python3* executable after the install.

The only other thing I can think of is to create the Codestar project, edit codebuild to use Ubuntu and then install everything (just like I did locally), but there is no way to do that from within Codestar and I feel like that may bring me down a rabbit hole and that's hardly automated. Is there a way to make that configuration as code from within my project?

EDIT Attempting to build and install Python 3.6 from source works, but then when trying to install Pip, I get errors saying SSL was not installed. And when looking back at the build logs, it seems other "bits" were not installed as well.

So, my questions here are:

How do I get Python 3.6 into a Codebuild environment provisioned from a Codestar project?
Should I continue trying to build it from source or make the switch to the Ubuntu environment?
How can I automatically configure the image / environment within my code/project?

EDIT 1 For anyone else, my complete buildspec.yml for installing and using Python3.6 is below. Note, it keeps everything as quiet as possible in order to reduce the log messages, reduce Cloudwatch cost and speed up the process. I ended up shaving about 90 seconds off the whole process by doing that (installing Python and building my app). Since CodeBuild charges based on time spent, this is crucial.

version: 0.2

phases:
  install:
    commands:
      - yum -qye 0 update
      - yum -qye 0 groupinstall development
      - yum -y install python-devel
      - yum -qye 0 install libxml2-devel libxslt-devel libjpeg-devel zlib-devel libpng-devel openssl-devel sqlite-devel
      - export HOME_DIR=`pwd`
      # I would recommend hosting the tarball in an uncompressed format on S3 in order to speed up the download and decompression
      - wget --no-verbose https://www.python.org/ftp/python/3.6.1/Python-3.6.1.tgz
      - tar -xzf Python-3.6.1.tgz
      - cd Python-3.6.1
      - ./configure -q --enable-loadable-sqlite-extensions
      - make --silent -j2
      - make altinstall --silent
      - cd $HOME_DIR
      - rm Python-3.6.1.tgz
      - rm -rf Python-3.6.1/
      - ln -s /usr/local/bin/python3.6 /usr/bin/python3
      - python3 -m pip install virtualenv
      - pip3 install -U nltk
  pre_build:
    commands:
      - cd $HOME_DIR
      # Start a virtualenv and activate
      - virtualenv -p /usr/bin/python3 $VIRTUAL_ENV_DIR_NAME
      - source $VIRTUAL_ENV_DIR_NAME/bin/activate
      - $VIRTUAL_ENV_DIR_NAME/bin/pip3.6 install nltk
      # If you plan to use any separate resources on Codecommit, you need to configure git
      - git config --global credential.helper '!aws codecommit credential-helper $@'
      - git config --global credential.UseHttpPath true
      # git clone whatever you need
  build:
    commands:
      - cd $HOME_DIR
      - mv $VIRTUAL_ENV/lib/python3.6/site-packages/* .
      - aws cloudformation package --template template.yml --s3-bucket $S3_BUCKET --output-template template-export.json
artifacts:
  type: zip
  files:
    - template-export.json

Solution

This is what my buildspec.yml looks like. Notice that the python3.6 version is output in pre_build phase.

version: 0.2

phases:
  install:
    commands:
      - yum -y groupinstall development
      - yum -y install zlib-devel
      - wget https://www.python.org/ftp/python/3.6.0/Python-3.6.0.tar.xz
      - tar xJf Python-3.6.0.tar.xz
      - cd Python-3.6.0
      - ./configure
      - make
      - make install 
  pre_build:
    commands:
      - python3 -V
  ...

Another way, to go about this is to upload a Python3.6 docker image to ECR. You can set the option to use this ECR image to run your build.