AWS Lambda job runs twice when using waiter.wait

I'm using a lambda function to run AWS elastic transcoder.

The function works fine if I do the following:

def lambda_handler(event, context):

  transcoder = boto3.client('elastictranscoder', REGION_NAME)
  pipeline_id = get_pipeline(transcoder, PIPELINE)

  input_file = os.path.basename(event['Records'][0]['s3']['object']['key'])
  filename_without_extension = os.path.splitext(input_file)[0]

  job = transcoder.create_job(
      PipelineId=pipeline_id,
      Input={
          'Key': 'uploads/' + input_file,
          'FrameRate': 'auto',
          'Resolution': 'auto',
          'AspectRatio': 'auto',
          'Interlaced': 'auto',
          'Container' : 'auto'
      },
      Outputs=[{
          'Key': filename_without_extension + '.mp4',
          'PresetId': '1487545782241-6uy45r',
      }]
  )

However what I'd like to do is delete the original file once the transcoding is completed.

(Recommended by Deleting a file after transcoding with AWS Lambda and Elastic Transcoder)

I'm trying to use the waiter.wait() function, but if I add the following to the end of my code:

job_id = job['Job']['Id']
waiter = transcoder.get_waiter('job_complete')
waiter.wait(Id=job_id)

then my job gets resubmitted? Basically the same job reappears in Elastic Transcoder, but fails as the output file already exists. The cloud watch logs show the job being run again as part of the same lambda execution.

How can I use the waiter.wait method to let me know when the job is completed, without it being resubmitted?



You need to subscribe to the completion notification under your pipeline settings. The notification will have to start up a new lambda function to handle your custom logic.

The reason you are seeing the same jobs get submitted again is because if a lambda function execution fails (by calling the completion block with an error, times out, etc.) lambda will automatically retry the function with the same event twice after a back off period.