amazon-web-servicesgithub-actionsamazon-lightsail

AWS Lightsail: "You must create a container service before retrieving container registry login credentials."


I'm attempting to deploy to Lightsail from a GitHub Action. I have a container service created (despite the error message) but I'm getting an error saying I don't. I imagine this has to do with privileges, but I can't figure out what might be wrong.

Here is the entire config file, but I believe this is the only relevant stanza for this discussion:

      - name: Release to Amazon Lightsail
        env:
          CONTAINER_SERVICE_NAME: ${{ env.ENVIRONMENT }}-${{ env.SERVICE_NAME }}-cs
        run: |
          echo "Releasing to Amazon Lightsail"

          docker pull $ORG_NAME/$SERVICE_NAME:$GITHUB_SHA

          echo "Uploading docker image to $CONTAINER_SERVICE_NAME"

          # upload the docker image for this pipeline
          aws --debug lightsail push-container-image \
            --service-name $CONTAINER_SERVICE_NAME  \
            --label ${{ env.SERVICE_NAME }}-latest  \
            --image $ORG_NAME/$SERVICE_NAME:$GITHUB_SHA

Here is the error:

Run echo "Releasing to Amazon Lightsail"
Releasing to Amazon Lightsail
61388d167c4340ec7054e7e7a64bcd897e407a9d: Pulling from ***/slackbot
[ lots of pulling and downloading ]
Digest: sha256:0d4f0cce97751a1f4ef5dfc5731ad09c2d7762f3c307215269cffccbdb655d79
Status: Downloaded newer image for ***/slackbot:61388d167c4340ec7054e7e7a64bcd897e407a9d
docker.io/***/slackbot:61388d167c4340ec7054e7e7a64bcd897e407a9d
Uploading docker image to production-slackbot-cs
2023-07-14 22:10:53,018 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.13.0 Python/3.11.4 Linux/5.15.0-1041-azure exe/x86_64.ubuntu.22
2023-07-14 22:10:53,018 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['--debug', 'lightsail', 'push-container-image', '--service-name', 'production-slackbot-cs', '--label', 'slackbot-latest', '--image', '***/slackbot:61388d167c4340ec7054e7e7a64bcd897e407a9d']
2023-07-14 22:10:54,466 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_s3 at 0x7fc0365e0360>
2023-07-14 22:10:54,467 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_ddb at 0x7fc036919b20>
2023-07-14 22:10:54,467 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.configure.configure.ConfigureCommand'>>
2023-07-14 22:10:54,467 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7fc036d77e20>
2023-07-14 22:10:54,467 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7fc036d9d4e0>
2023-07-14 22:10:54,467 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function alias_opsworks_cm at 0x7fc0365e2de0>
2023-07-14 22:10:54,467 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_history_commands at 0x7fc036970540>
2023-07-14 22:10:54,467 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.devcommands.CLIDevCommand'>>
2023-07-14 22:10:54,467 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_waiters at 0x7fc0365e2ca0>
2023-07-14 22:10:54,467 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x7fc035f9d6d0>>
2023-07-14 22:10:54,467 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.13.0/dist/awscli/data/cli.json
2023-07-14 22:10:54,469 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_types at 0x7fc03651df80>
2023-07-14 22:10:54,469 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function no_sign_request at 0x7fc03651e2a0>
2023-07-14 22:10:54,469 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_verify_ssl at 0x7fc03651e200>
2023-07-14 22:10:54,469 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_read_timeout at 0x7fc03651e3e0>
2023-07-14 22:10:54,470 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_connect_timeout at 0x7fc03651e340>
2023-07-14 22:10:54,470 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <built-in method update of dict object at 0x7fc035f928c0>
2023-07-14 22:10:54,470 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.13.0 Python/3.11.4 Linux/5.15.0-1041-azure exe/x86_64.ubuntu.22 prompt/off
2023-07-14 22:10:54,470 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['--debug', 'lightsail', 'push-container-image', '--service-name', 'production-slackbot-cs', '--label', 'slackbot-latest', '--image', '***/slackbot:61388d167c4340ec7054e7e7a64bcd897e407a9d']
2023-07-14 22:10:54,470 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_timestamp_parser at 0x7fc0365e0cc0>
2023-07-14 22:10:54,470 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function register_uri_param_handler at 0x7fc0375f9ee0>
2023-07-14 22:10:54,471 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_binary_formatter at 0x7fc035f602c0>
2023-07-14 22:10:54,471 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function no_pager_handler at 0x7fc0375207c0>
2023-07-14 22:10:54,471 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x7fc037115800>
2023-07-14 22:10:54,567 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
2023-07-14 22:10:54,569 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x7fc036950e00>
2023-07-14 22:10:54,569 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_json_file_cache at 0x7fc036dfdda0>
2023-07-14 22:10:54,628 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.13.0/dist/awscli/botocore/data/lightsail/2016-11-28/service-2.json
2023-07-14 22:10:54,658 - MainThread - botocore.hooks - DEBUG - Event building-command-table.lightsail: calling handler <function inject_commands at 0x7fc035f605e0>
2023-07-14 22:10:54,658 - MainThread - botocore.hooks - DEBUG - Event building-command-table.lightsail: calling handler <function add_waiters at 0x7fc0365e2ca0>
2023-07-14 22:10:54,668 - MainThread - botocore.hooks - DEBUG - Event building-command-table.lightsail: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x7fc035f9d6d0>>
2023-07-14 22:10:54,669 - MainThread - botocore.hooks - DEBUG - Event building-command-table.lightsail_push-container-image: calling handler <function add_waiters at 0x7fc0365e2ca0>
2023-07-14 22:10:54,669 - MainThread - botocore.hooks - DEBUG - Event building-command-table.lightsail_push-container-image: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x7fc035f9d6d0>>
2023-07-14 22:10:54,670 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.push-container-image.service-name: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fc0381e8090>
2023-07-14 22:10:54,670 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.push-container-image: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7fc03753e6d0>
2023-07-14 22:10:54,670 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.push-container-image.image: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fc0381e8090>
2023-07-14 22:10:54,670 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.push-container-image: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7fc03753e6d0>
2023-07-14 22:10:54,670 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.push-container-image.label: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fc0381e8090>
2023-07-14 22:10:54,670 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.push-container-image: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7fc03753e6d0>
2023/07/14 22:10:54 DEBUG: Request Signature:
---[ CANONICAL STRING  ]-----------------------------
POST
/

content-length:2
content-type:application/x-amz-json-1.1
host:lightsail.***.amazonaws.com
x-amz-date:20230714T221054Z
x-amz-security-token:***
x-amz-target:Lightsail_20161128.GetContainerAPIMetadata

content-length;content-type;host;x-amz-date;x-amz-security-token;x-amz-target
44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
---[ STRING TO SIGN ]--------------------------------
AWS4-HMAC-SHA256
20230714T221054Z
20230714/***/lightsail/aws4_request
a0d61a57603c598459025912f6107fa80f82c6abf01fcf62cdfb0d607fb8f221
-----------------------------------------------------
2023/07/14 22:10:54 DEBUG: Request lightsail/GetContainerAPIMetadata Details:
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: lightsail.***.amazonaws.com
User-Agent: aws-sdk-go/1.44.39 (go1.18.1; linux; amd64) lightsailctl/v1.0.4 (go1.18.1; linux; amd64)
Content-Length: 2
Authorization: AWS4-HMAC-SHA256 Credential=***/20230714/***/lightsail/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date;x-amz-security-token;x-amz-target, Signature=59e575cba84116c94392b5a472a15a699d2ebc2ae5ff54a3abaf808c2bf51282
Content-Type: application/x-amz-json-1.1
X-Amz-Date: 20230714T221054Z
X-Amz-Security-Token: ***
X-Amz-Target: Lightsail_20161128.GetContainerAPIMetadata
Accept-Encoding: gzip

{}
-----------------------------------------------------
2023/07/14 22:10:55 DEBUG: Response lightsail/GetContainerAPIMetadata Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Content-Length: 61
Connection: keep-alive
Content-Type: application/x-amz-json-1.1
Date: Fri, 14 Jul 2023 22:10:55 GMT
Server: Server
X-Amzn-Requestid: 8ba40949-9a3d-4cb9-93e0-0887f28c7200


-----------------------------------------------------
2023/07/14 22:10:55 {"metadata":[{"name":"lightsailctlVersion","value":"1.0.0"}]}
2023/07/14 22:10:55 DEBUG: Request Signature:
---[ CANONICAL STRING  ]-----------------------------
POST
/

content-length:2
content-type:application/x-amz-json-1.1
host:lightsail.***.amazonaws.com
x-amz-date:20230714T221055Z
x-amz-security-token:***
x-amz-target:Lightsail_20161128.CreateContainerServiceRegistryLogin

content-length;content-type;host;x-amz-date;x-amz-security-token;x-amz-target
44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
---[ STRING TO SIGN ]--------------------------------
AWS4-HMAC-SHA256
20230714T221055Z
20230714/***/lightsail/aws4_request
fa25e2fad9231a25f2ecfe7922efd5c8ceaa1dfdf1316d822039318ab3febb69
-----------------------------------------------------
2023/07/14 22:10:55 DEBUG: Request lightsail/CreateContainerServiceRegistryLogin Details:
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: lightsail.***.amazonaws.com
User-Agent: aws-sdk-go/1.44.39 (go1.18.1; linux; amd64) lightsailctl/v1.0.4 (go1.18.1; linux; amd64)
Content-Length: 2
Authorization: AWS4-HMAC-SHA256 Credential=***/20230714/***/lightsail/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date;x-amz-security-token;x-amz-target, Signature=a97ac5132d3195815d2b1d5ebe18a198e957b2e1acad403e02cdeeee8c7b6d71
Content-Type: application/x-amz-json-1.1
X-Amz-Date: 20230714T221055Z
X-Amz-Security-Token: ***
X-Amz-Target: Lightsail_20161128.CreateContainerServiceRegistryLogin
Accept-Encoding: gzip

{}
-----------------------------------------------------
2023/07/14 22:10:55 DEBUG: Response lightsail/CreateContainerServiceRegistryLogin Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 400 Bad Request
Content-Length: 138
Connection: keep-alive
Content-Type: application/x-amz-json-1.1
Date: Fri, 14 Jul 2023 22:10:55 GMT
Server: Server
X-Amzn-Requestid: 94ad92e5-de73-4e00-aff1-a99a8ca74b45


-----------------------------------------------------
2023/07/14 22:10:55 {"__type":"InvalidInputException","message":"You must create a container service before retrieving container registry login credentials."}
InvalidInputException: You must create a container service before retrieving container registry login credentials.
{
  RespMetadata: {
    StatusCode: 400,
    RequestID: "94ad92e5-de73-4e00-aff1-a99a8ca74b45"
  },
  Message_: "You must create a container service before retrieving container registry login credentials."
}
2023-07-14 22:10:55,355 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "awscli/clidriver.py", line 460, in main
  File "awscli/clidriver.py", line 595, in __call__
  File "awscli/customizations/commands.py", line 205, in __call__
  File "awscli/customizations/lightsail/push_container_image.py", line 65, in _run_main
  File "subprocess.py", line 571, in run
subprocess.CalledProcessError: Command '['lightsailctl', '--plugin', '--input-stdin']' returned non-zero exit status 1.

Command '['lightsailctl', '--plugin', '--input-stdin']' returned non-zero exit status 1.
Error: Process completed with exit code 255.

Here are my privileges:

{
            "Effect": "Allow",
            "Action": [
                "lightsail:GetContainerImages",
                "lightsail:GetContainerAPIMetadata",
                "lightsail:CreateContainerService",
                "lightsail:CreateContainerServiceRegistryLogin",
                "lightsail:GetContainerServices",
                "lightsail:GetContainerServiceDeployments",
                "lightsail:GetContainerServicePowers"
            ],
            "Resource": "*"
        },
{
            "Effect": "Allow",
            "Action": [
                "lightsail:CreateContainerServiceDeployment",
                "lightsail:DeleteContainerService",
                "lightsail:RegisterContainerImage",
                "lightsail:UpdateContainerService"
            ],
            "Resource": [my ARN]
        },
}

Anyway, we're stumped. :) Thanks!


Solution

  • The issue was resolved by changing the curl target for the lightsailctl binary from s3.us-west-2.amazonaws.com to s3.us-east-1.amazonaws.com in the deploy-service-production's run command.