TooManyRequestsException while calling mediaconvert for updating job status in UI using javascript SDK

The application is developed using React Js as a frontend and nodeJs as backend.The application is deployed in AWS EKS cluster.The application is like a videoportal where users can upload a video and the video is processed using AWS mediaconvert and store the processed files in S3. Even after the job is completed in mediaconvert,it is not updating the status in UI.It throws TooManyRequestsException.

TooManyRequestsException

The TooManyRequestsException exception you are encountering is as a result of API throttling while trying to poll job status' [1]. With the current approach you would need to catch for the TooManyRequestsException and try the poll with a back off threshold [2].

The most scalable solution is to use CloudWatch Events and track jobs based the STATUS_UPDATE, COMPLETE and ERROR status [3].

Example of the Event Pattern

{
  "source": [
    "aws.mediaconvert"
  ],
  "detail-type": [
    "MediaConvert Job State Change"
  ],
  "detail": {
    "status": [
      "STATUS_UPDATE"
    ]
  }
}

Example of what the Status Update event payload looks like:

{
    "version": "0",
    "id": "ABC",
    "detail-type": "MediaConvert Job State Change",
    "source": "aws.mediaconvert",
    "account": "111122223333 ",
    "time": "2021-02-18T17:52:32Z",
    "region": "us-west-2",
    "resources": [
        "arn:aws:mediaconvert:us-west-2:111122223333 :jobs/1613670689802-emcngz"
    ],
    "detail": {
        "timestamp": 1613670752653,
        "accountId": "111122223333 ",
        "queue": "arn:aws:mediaconvert:us-west-2:111122223333 :queues/Default",
        "jobId": "1613670689802-emcngz",
        "status": "STATUS_UPDATE",
        "userMetadata": {},
        "framesDecoded": 2024,
        "jobProgress": {
            "phaseProgress": {
                "PROBING": {
                    "status": "COMPLETE",
                    "percentComplete": 100
                },
                "TRANSCODING": {
                    "status": "PROGRESSING",
                    "percentComplete": 2
                },
                "UPLOADING": {
                    "status": "PENDING",
                    "percentComplete": 0
                }
            },
            "jobPercentComplete": 7,
            "currentPhase": "TRANSCODING",
            "retryCount": 0
        }
    }
}

MediaConvert provides granular percentages per phases the job is in (probing input, transcoding, and uploading outputs) as well as an overall percentage. The one that is displayed on the MediaConvert Console UI is the jobPercentComplete, and would probably be the one you want to capture.

The CloudWatch Event rule can potentially be setup to use an SQS Queue as its destination. In this case the EKS hosted application can then poll the queue. Alternatively, the rule can be setup to invoke a lambda function. In this case, you can have NodeJS code that processes the event.

1. AWS Elemental MediaConvert endpoints and quotas: https://docs.aws.amazon.com/general/latest/gr/mediaconvert.html

2. Error retries and exponential backoff in AWS : https://docs.aws.amazon.com/general/latest/gr/api-retries.html

3. Monitoring AWS Elemental MediaConvert : https://docs.aws.amazon.com/mediaconvert/latest/ug/monitoring-overview.html