python-3.x google-cloud-platform google-app-engine dockerfile

Permissions error when deploying to Google App Engine using flexible environment and Dockerfile

I want to deploy a python hello-world service by building an image using a Dockerfile and deploying to Google Cloud App Engine. These are the files I’m using for this purpose:

# app.py

#!/usr/bin/python

import time
from flask import Flask
app = Flask(__name__)

START = time.time()

def elapsed():
    running = time.time() - START
    minutes, seconds = divmod(running, 60)
    hours, minutes = divmod(minutes, 60)
    return "%d:%02d:%02d" % (hours, minutes, seconds)

@app.route('/')
def root():
    return "Hello World (Python)! (up %s)\n" % elapsed()

if __name__ == "__main__":
    app.run(debug=True, host="0.0.0.0", port=8080)

# requirements.txt

flask

# Dockerfile

FROM python:3-alpine

WORKDIR /service
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . ./

EXPOSE 8080

ENTRYPOINT ["python3", "app.py"]

# app. yaml

runtime: custom
env: flex

service: py-hello

This code runs successfully when running it using docker build and docker run commands.

When deploying this application to Google App Engine I get an error:

$ gcloud app deploy

Services to deploy:

descriptor:                  [/Users/carlostomas/Workspace/gae-py-hello/app.yaml]
source:                      [/Users/carlostomas/Workspace/gae-py-hello]
target project:              [intick-rfq-dev]
target service:              [py-hello]
target version:              [20240623t113720]
target url:                  [https://py-hello-dot-intick-rfq-dev.nw.r.appspot.com]
target service account:      [sa-intick-rfq-dev-tf@intick-rfq-dev.iam.gserviceaccount.com]


Do you want to continue (Y/n)?  y

Beginning deployment of service [py-hello]...
╔════════════════════════════════════════════════════════════╗
╠═ Uploading 4 files to Google Cloud Storage                ═╣
╚════════════════════════════════════════════════════════════╝
File upload done.
Updating service [py-hello] (this may take several minutes)...done.
Waiting for build to complete. Polling interval: 1 second(s).
------------------------------------------------------------------------- REMOTE BUILD OUTPUT --------------------------------------------------------------------------
starting build "68970ce5-9ea1-4a8b-ad20-43b97eccda25"

FETCHSOURCE
BUILD
Starting Step #0 - "fetcher"
Step #0 - "fetcher": Already have image (with digest): gcr.io/cloud-builders/gcs-fetcher
Step #0 - "fetcher": Fetching manifest gs://staging.intick-rfq-dev.appspot.com/ae/3e87d960-7d7d-4a44-a7bf-1b0af0eac2bb/manifest.json.
Step #0 - "fetcher": Processing 4 files.
Step #0 - "fetcher": ******************************************************
Step #0 - "fetcher": Status:                      SUCCESS
Step #0 - "fetcher": Started:                     2024-06-23T09:37:32Z
Step #0 - "fetcher": Completed:                   2024-06-23T09:37:34Z
Step #0 - "fetcher": Requested workers:    200
Step #0 - "fetcher": Actual workers:         4
Step #0 - "fetcher": Total files:            4
Step #0 - "fetcher": Total retries:          0
Step #0 - "fetcher": GCS timeouts:           0
Step #0 - "fetcher": MiB downloaded:         0.00 MiB
Step #0 - "fetcher": MiB/s throughput:       0.00 MiB/s
Step #0 - "fetcher": Time for manifest:    950.67 ms
Step #0 - "fetcher": Total time:             1.88 s
Step #0 - "fetcher": ******************************************************
Finished Step #0 - "fetcher"
Starting Step #1
Step #1: Already have image (with digest): gcr.io/cloud-builders/docker
Step #1: Sending build context to Docker daemon   5.12kB
Step #1: Step 1/7 : FROM python:3-alpine
Step #1: 3-alpine: Pulling from library/python
Step #1: ec99f8b99825: Pulling fs layer
Step #1: a68bf89b0030: Pulling fs layer
Step #1: dc4556e82255: Pulling fs layer
Step #1: 32e2ec03db6e: Pulling fs layer
Step #1: eb2ccc024fc3: Pulling fs layer
Step #1: eb2ccc024fc3: Waiting
Step #1: 32e2ec03db6e: Waiting
Step #1: a68bf89b0030: Verifying Checksum
Step #1: a68bf89b0030: Download complete
Step #1: ec99f8b99825: Download complete
Step #1: dc4556e82255: Verifying Checksum
Step #1: 32e2ec03db6e: Verifying Checksum
Step #1: 32e2ec03db6e: Download complete
Step #1: dc4556e82255: Download complete
Step #1: ec99f8b99825: Pull complete
Step #1: eb2ccc024fc3: Verifying Checksum
Step #1: eb2ccc024fc3: Download complete
Step #1: a68bf89b0030: Pull complete
Step #1: dc4556e82255: Pull complete
Step #1: 32e2ec03db6e: Pull complete
Step #1: eb2ccc024fc3: Pull complete
Step #1: Digest: sha256:dc095966439c68283a01dde5e5bc9819ba24b28037dddd64ea224bf7aafc0c82
Step #1: Status: Downloaded newer image for python:3-alpine
Step #1:  ---> 5d3a5c7fea1e
Step #1: Step 2/7 : WORKDIR /service
Step #1:  ---> Running in 629ca24729e4
Step #1: Removing intermediate container 629ca24729e4
Step #1:  ---> 749449f276aa
Step #1: Step 3/7 : COPY requirements.txt .
Step #1:  ---> ebefde10db6e
Step #1: Step 4/7 : RUN pip install -r requirements.txt
Step #1:  ---> Running in 0f7146aae859
Step #1: Collecting flask (from -r requirements.txt (line 1))
Step #1:   Downloading flask-3.0.3-py3-none-any.whl.metadata (3.2 kB)
Step #1: Collecting Werkzeug>=3.0.0 (from flask->-r requirements.txt (line 1))
Step #1:   Downloading werkzeug-3.0.3-py3-none-any.whl.metadata (3.7 kB)
Step #1: Collecting Jinja2>=3.1.2 (from flask->-r requirements.txt (line 1))
Step #1:   Downloading jinja2-3.1.4-py3-none-any.whl.metadata (2.6 kB)
Step #1: Collecting itsdangerous>=2.1.2 (from flask->-r requirements.txt (line 1))
Step #1:   Downloading itsdangerous-2.2.0-py3-none-any.whl.metadata (1.9 kB)
Step #1: Collecting click>=8.1.3 (from flask->-r requirements.txt (line 1))
Step #1:   Downloading click-8.1.7-py3-none-any.whl.metadata (3.0 kB)
Step #1: Collecting blinker>=1.6.2 (from flask->-r requirements.txt (line 1))
Step #1:   Downloading blinker-1.8.2-py3-none-any.whl.metadata (1.6 kB)
Step #1: Collecting MarkupSafe>=2.0 (from Jinja2>=3.1.2->flask->-r requirements.txt (line 1))
Step #1:   Downloading MarkupSafe-2.1.5-cp312-cp312-musllinux_1_1_x86_64.whl.metadata (3.0 kB)
Step #1: Downloading flask-3.0.3-py3-none-any.whl (101 kB)
Step #1:    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 101.7/101.7 kB 5.7 MB/s eta 0:00:00
Step #1: Downloading blinker-1.8.2-py3-none-any.whl (9.5 kB)
Step #1: Downloading click-8.1.7-py3-none-any.whl (97 kB)
Step #1:    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 97.9/97.9 kB 15.2 MB/s eta 0:00:00
Step #1: Downloading itsdangerous-2.2.0-py3-none-any.whl (16 kB)
Step #1: Downloading jinja2-3.1.4-py3-none-any.whl (133 kB)
Step #1:    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 133.3/133.3 kB 13.2 MB/s eta 0:00:00
Step #1: Downloading werkzeug-3.0.3-py3-none-any.whl (227 kB)
Step #1:    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 227.3/227.3 kB 31.8 MB/s eta 0:00:00
Step #1: Downloading MarkupSafe-2.1.5-cp312-cp312-musllinux_1_1_x86_64.whl (33 kB)
Step #1: Installing collected packages: MarkupSafe, itsdangerous, click, blinker, Werkzeug, Jinja2, flask
Step #1: Successfully installed Jinja2-3.1.4 MarkupSafe-2.1.5 Werkzeug-3.0.3 blinker-1.8.2 click-8.1.7 flask-3.0.3 itsdangerous-2.2.0
Step #1: WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Step #1:
Step #1: [notice] A new release of pip is available: 24.0 -> 24.1
Step #1: [notice] To update, run: pip install --upgrade pip
Step #1: Removing intermediate container 0f7146aae859
Step #1:  ---> a48c5a156824
Step #1: Step 5/7 : COPY . ./
Step #1:  ---> 2b409b2cf985
Step #1: Step 6/7 : EXPOSE 8080
Step #1:  ---> Running in d6bfba38f90f
Step #1: Removing intermediate container d6bfba38f90f
Step #1:  ---> 37d35496f2e7
Step #1: Step 7/7 : ENTRYPOINT ["python3", "app.py"]
Step #1:  ---> Running in 5af754f41b39
Step #1: Removing intermediate container 5af754f41b39
Step #1:  ---> d7c26a000a2b
Step #1: Successfully built d7c26a000a2b
Step #1: Successfully tagged eu.gcr.io/intick-rfq-dev/appengine/py-hello.20240623t113720:latest
Finished Step #1
PUSH
Pushing eu.gcr.io/intick-rfq-dev/appengine/py-hello.20240623t113720
The push refers to repository [eu.gcr.io/intick-rfq-dev/appengine/py-hello.20240623t113720]
0a46395f91fe: Preparing
3ed896076f1d: Preparing
e033cfdc4107: Preparing
534fdafdd34c: Preparing
da17acc3bd72: Preparing
d99d106fe90f: Preparing
407fd3c96102: Preparing
699ede46ccf2: Preparing
94e5f06ff8e3: Preparing
d99d106fe90f: Waiting
407fd3c96102: Waiting
699ede46ccf2: Waiting
94e5f06ff8e3: Waiting
da17acc3bd72: Layer already exists
d99d106fe90f: Layer already exists
407fd3c96102: Layer already exists
699ede46ccf2: Layer already exists
94e5f06ff8e3: Layer already exists
0a46395f91fe: Pushed
e033cfdc4107: Pushed
534fdafdd34c: Pushed
3ed896076f1d: Pushed
latest: digest: sha256:e3539e8bfaab427b6ef99de35ccd6f3f71f9b1b0f21698a85b528a2e08d858c2 size: 2200
DONE
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Updating service [py-hello] (this may take several minutes)...failed.
ERROR: (gcloud.app.deploy) Error Response: [7] The App Engine appspot and App Engine flexible environment service accounts must have permissions on the image [eu.gcr.io/intick-rfq-dev/appengine/py-hello.20240623t113720]. Please check that the App Engine default service account has the [Storage Object Viewer] role and the App Engine  Flexible service account has the App Engine Flexible Environment Service Agent role

The image eu.gcr.io/intick-rfq-dev/appengine/py-hello.20240623t113720:latest has been successfully pulled to the Google Container Registry. I can even pull this image to my local machine and run a docker run command and the service runs in a docker containter.

This error only happens when trying to deploy the application by building a Dockerfile and using the flexible environment (env: flex). I have no issues when deploying other applications (nodejs, python, java) using the standard environemnt and no Dockerfile.

These are the roles associated to the App Engine default and flexible environment service accounts:

$ gcloud projects get-iam-policy intick-rfq-dev \
    --flatten="bindings[].members" \
    --format='table(bindings.role)' \
    --filter="bindings.members:intick-rfq-dev@appspot.gserviceaccount.com"
ROLE
roles/appengineflex.serviceAgent
roles/editor
roles/secretmanager.secretAccessor
roles/storage.objectViewer

$ gcloud projects get-iam-policy intick-rfq-dev \
    --flatten="bindings[].members" \
    --format='table(bindings.role)' \
    --filter="bindings.members:service-169683775925@gae-api-prod.google.com.iam.gserviceaccount.com"
ROLE
roles/appengineflex.serviceAgent

It looks that the App Engine service accounts have the right roles associated to them.

According to the troubleshoot App Engine errors page: https://cloud.google.com/appengine/docs/flexible/troubleshooting#service-account-permissions

There are two potential causes to this error:

The default App Engine service account does not have the Storage Object Viewer (roles/storage.objectViewer) role.
Your project has a VPC Service Perimeter which limits access to the Cloud Storage API using access levels.

The first one can't be the reason because the service account has that role.

The second one I don't know how to deal with that. I'm a newbie in Google Cloud and don't know how to grant access to Cloud Storage API in the VPC Service Perimeter.

Can someone help me how to deal with VPC Service Perimeter and access levels regarding the Cloud Storage API to solve this problem? Or maybe the error can come from a different reason.

Solution

Thanks for your feedback @DazWilkin. From the documentation, it looks like App Engine uses a default service account created with the project application:

${PROJECT}@@appspot.gserviceaccount.com

What was happening was that I had a different account associated to the project:

$ cloud app describe
authDomain: gmail.com
codeBucket: staging.{PROJECT}.appspot.com
databaseType: CLOUD_DATASTORE_COMPATIBILITY
defaultBucket: {PROJECT}.appspot.com
defaultHostname: {PROJECT}.nw.r.appspot.com
featureSettings:
  splitHealthChecks: true
  useContainerOptimizedOs: true
gcrDomain: eu.gcr.io
id: {PROJECT}
locationId: europe-west2
name: apps/{PROJECT}
serviceAccount: sa-{PROJECT}-tf@{PROJECT}.iam.gserviceaccount.com
servingStatus: SERVING

We can see that I had assigned a different service account to the application. So the solution is to reassign the default service account with the overgranted 'Editor' permissions:

$ gcloud app update --service-account=${PROJECT}@appspot.gserviceaccount.com

$ cloud app describe
authDomain: gmail.com
codeBucket: staging.{PROJECT}.appspot.com
databaseType: CLOUD_DATASTORE_COMPATIBILITY
defaultBucket: {PROJECT}.appspot.com
defaultHostname: {PROJECT}.nw.r.appspot.com
featureSettings:
  splitHealthChecks: true
  useContainerOptimizedOs: true
gcrDomain: eu.gcr.io
id: {PROJECT}
locationId: europe-west2
name: apps/{PROJECT}
serviceAccount: {PROJECT}@appspot.gserviceaccount.com
servingStatus: SERVING

And now the application deploys successfully when running gcloud app deploy.

As a best practice, permissions should be restricted to the default App Engine service account and only grant those required to deploy my application.