Google App Engine Flexible deploy throwing ERROR: (gcloud.app.deploy) HttpError accessing

Hi I am getting this error when deploying nodejs application to flexible engine. I am unable to figure out where the issue is happening.

The error message I am getting

ERROR: (gcloud.app.deploy) HttpError accessing <https://appengine.googleapis.com/v1/apps/project-id/services/testws/versions?alt=json>: response: <{'vary': 'Origin, X-Origin, Referer', 'content-type': 'application/json; charset=UTF-8', 'date': 'Fri, 26 Mar 2021 20:37:27 GMT', 'server': 'ESF', 'cache-control': 'private', 'x-xss-protection': '0', 'x-frame-options': 'SAMEORIGIN', 'x-content-type-options': 'nosniff', 'alt-svc': 'h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"', 'transfer-encoding': 'chunked', 'status': '500', 'content-length': '109', '-content-encoding': 'gzip'}>, content <{
  "error": {
    "code": 500,
    "message": "Internal error encountered.",
    "status": "INTERNAL"
  }
}
>
This may be due to network connectivity issues. Please check your network settings, and the status of the service you are trying to reach.

app.yaml

runtime: nodejs
env: flex
service: testws
manual_scaling:
  instances: 1
network:
  session_affinity: true
resources:
  cpu: 1
  memory_gb: 1
  disk_size_gb: 2
env_variables:
  KEY: "dev"
  PRIVATE_KEY_URL: "key"
skip_files:
  - ^node_modules/.*$

Gcloud SDK version

Google Cloud SDK 333.0.0
bq 2.0.65
core 2021.03.19
gsutil 4.60
kubectl 1.17.17

I have added the debug logs by running gcloud app deploy --verbosity="debug"

debug logs

>
This may be due to network connectivity issues. Please check your network settings, and the status of the service you are trying to reach.
Traceback (most recent call last):
  File "/Users/shashi/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 982, in Execute
    resources = calliope_command.Run(cli=self, args=args)
  File "/Users/shashi/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 809, in Run
    resources = command_instance.Run(args)
  File "/Users/shashi/google-cloud-sdk/lib/surface/app/deploy.py", line 130, in Run
    use_legacy_apis=args.use_legacy_apis)
  File "/Users/shashi/google-cloud-sdk/lib/googlecloudsdk/command_lib/app/deploy_util.py", line 678, in RunDeploy
    ignore_file=args.ignore_file)
  File "/Users/shashi/google-cloud-sdk/lib/googlecloudsdk/command_lib/app/deploy_util.py", line 455, in Deploy
    extra_config_settings)
  File "/Users/shashi/google-cloud-sdk/lib/googlecloudsdk/api_lib/app/appengine_api_client.py", line 172, in DeployService
    extra_config_settings)
  File "/Users/shashi/google-cloud-sdk/lib/googlecloudsdk/api_lib/app/appengine_api_client.py", line 249, in _CreateVersion
    return self.client.apps_services_versions.Create(create_request)
  File "/Users/shashi/google-cloud-sdk/lib/googlecloudsdk/third_party/apis/appengine/v1/appengine_v1_client.py", line 830, in Create
    config, request, global_params=global_params)
  File "/Users/shashi/google-cloud-sdk/lib/third_party/apitools/base/py/base_api.py", line 729, in _RunMethod
    http, http_request, **opts)
  File "/Users/shashi/google-cloud-sdk/lib/third_party/apitools/base/py/http_wrapper.py", line 350, in MakeRequest
    check_response_func=check_response_func)
  File "/Users/shashi/google-cloud-sdk/lib/third_party/apitools/base/py/http_wrapper.py", line 406, in _MakeRequestNoRetry
    check_response_func(response)
  File "/Users/shashi/google-cloud-sdk/lib/googlecloudsdk/api_lib/util/apis.py", line 267, in _CheckResponseForApiEnablement
    http_wrapper.CheckResponse(response)
  File "/Users/shashi/google-cloud-sdk/lib/third_party/apitools/base/py/http_wrapper.py", line 223, in CheckResponse
    raise exceptions.BadStatusCodeError.FromResponse(response)
apitools.base.py.exceptions.BadStatusCodeError: HttpError accessing <https://appengine.googleapis.com/v1/apps/project-id/services/awsockets/versions?alt=json>: response: <{'vary': 'Origin, X-Origin, Referer', 'content-type': 'application/json; charset=UTF-8', 'date': 'Sun, 28 Mar 2021 22:29:25 GMT', 'server': 'ESF', 'cache-control': 'private', 'x-xss-protection': '0', 'x-frame-options': 'SAMEORIGIN', 'x-content-type-options': 'nosniff', 'alt-svc': 'h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"', 'transfer-encoding': 'chunked', 'status': '500', 'content-length': '109', '-content-encoding': 'gzip'}>, content <{
  "error": {
    "code": 500,
    "message": "Internal error encountered.",
    "status": "INTERNAL"
  }
}
>

I found the reason for the error and the solution to fix it if anyone is facing this issue.

Reason - The App engine Flexible Service Account was accidentally deleted from the google cloud project. As mentioned in this link - service-account

The App Engine flexible environment service agent has the App Engine Flexible Environment Service Agent role. The role includes a set of permissions needed by Node.js flexible environment to manage your flexible environment apps. For example, this role includes permissions to perform the following tasks:

 1. Deploying a new version.
 2. Stopping or deleting existing versions.
 3. Automatic weekly restarts and system updates.

Solution - Add a service account by going to IAM and adding an account of name mentioned below with role App Engine Flexible Environment Service Agent. As mentioned in this link - restore-service-account

service-[YOUR_PROJECT_NUMBER]@gae-api-prod.google.com.iam.gserviceaccount.com

I have raised an issue with google tracker to change the error message. You can track the issue here - issue-tracker

Update - adding the below image for further clarity. if you try to add the deleted account, it will be suggested by the system.