Invoke REST API times out for Approval and Checks on Azure Build Agent pool

I have a lambda endpoint in AWS that spins up an ec2 instance that create a build agent, and then waits for the agent to be connected in azure before responding. This takes about 1.5 minutes, however when I run the pipeline the Invoke REST API approval/check times out at 60 seconds:

============================================================================== 
Task         : Invoke REST API
Description  : Invoke a REST API as a part of your pipeline.
Version      : 1.220.0
Author       : Microsoft Corporation
Help URL     : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/http-rest-api
============================================================================== 

                Parsing expression: <TaskInputs[TaskInputs['connectedServiceNameSelector']]>
                TaskInputs
                [
                ..TaskInputs
                ..[
                ....'connectedServiceNameSelector'
                ..]
                ]
                Evaluating: TaskInputs[TaskInputs['connectedServiceNameSelector']]
                Evaluating indexer:
                ..Evaluating TaskInputs:
                ..=> Object
                ..Evaluating indexer:
                ....Evaluating TaskInputs:
                ....=> Object
                ....Evaluating String:
                ....=> 'connectedServiceNameSelector'
                ..=> 'connectedServiceName'
                => 'ce006752-214c-44f5-9c56-c96be6a8cc3d'
                Result: 'ce006752-214c-44f5-9c56-c96be6a8cc3d'

2024-11-29T10:11:08.6501699Z GET https://XXX.lambda-url.eu-west-2.on.aws/
                Response Code: 0
                Response: An error was encountered while processing request. Exception: 
Exception Message: The request timed out after 60 seconds as no response was received. (type TimeoutException)
                Evaluation of expression 'eq(root['statuscode'], 200)' failed.
                
Exception Message: Error parsing API response into a JSON object. (type DistributedTaskException)

In the approval and check, I have put the timeout at 1440 minutes:

So I am not sure why it is timing out after 60 seconds. Is there a way that I can either increase the timeout, or create another lambda endpoint which you can call to see if the agent is online?

What you should do turn your invoke REST API approval and check to a callback completion event: And then make your original lambda endpoint call another one with the InvocationType 'Event':

import boto3
import os
import logging
import json

logger = logging.getLogger()
logger.setLevel(logging.INFO)


def lambda_handler(event, context):
    plan_url = event['headers'].get('PlanUrl')
    project_id = event['headers'].get('ProjectId')
    hub_name = event['headers'].get('HubName')
    plan_id = event['headers'].get('PlanId')
    job_id = event['headers'].get('JobId')
    task_instance_id = event['headers'].get('TaskInstanceId')
    auth_token = event['headers'].get('AuthToken')
    
    # Do what you want to do here to set up the build server

    ec2_client = boto3.client('ec2', region_name=region)
    ec2_client.run_instances(
        LaunchTemplate={
            'LaunchTemplateId': launch_template_id,
            'Version': '$Latest'
        },
        MinCount=1,
        MaxCount=1,
        SubnetId=subnet_id,
    )

    lambda_client = boto3.client("lambda")
    lambda_client.invoke(
        FunctionName=os.environ['CALLBACK_FUNCTION_NAME'],
        InvocationType='Event',
        Payload=json.dumps({
                'PlanUrl': plan_url,
                'ProjectId': project_id,
                'HubName': hub_name,
                'PlanId': plan_id,
                'JobId': job_id,
                'TaskInstanceId': task_instance_id,
                'AuthToken': auth_token
        })
    )
    return {
        'statusCode': 200
    }

then that lambda endpoint can wait until the build agent has been created, and then call the azure devops callback api to complete the invoke REST API event:

import os
import time
import logging
import requests
from requests.auth import HTTPBasicAuth
import base64
import json


logger = logging.getLogger()
logger.setLevel(logging.INFO)

def lambda_handler(event, context):
    project_id = event.get('ProjectId')
    hub_name = event.get('HubName')
    plan_id = event.get('PlanId')
    job_id = event.get('JobId')
    task_instance_id = event.get('TaskInstanceId')
    auth_token = event.get('AuthToken')

    api_url = f"https://dev.azure.com/{organization}/_apis/distributedtask/pools/{pool_id}/agents?api-version=7.0"
    result = "succeded"
    ado_events_url = f"https://dev.azure.com/{organization}/{project_id}/_apis/distributedtask/hubs/{hub_name}/plans/{plan_id}/events?api-version=7.1-preview.1"
    headers = {
        'Accept': 'application/json',
        'Authorization': 'Basic '+str((base64.b64encode(bytes('ecs-ado-callback:'+auth_token, 'ascii'))), 'ascii')
    }
    body = {"name": "TaskCompleted", "taskId": task_instance_id, "jobId": job_id, "result": result}
    logger.info('ado post succeeded details:')
    logger.info(ado_events_url)
    logger.info(body)
    response = requests.request("POST", ado_events_url, headers=headers, json=body)
    logger.info("response from ADO:")
    logger.info(response.json())

    return {
        'statusCode': 200,
        'body': json.dumps({
            'name': 'TaskCompleted',
            'taskId': task_instance_id,
            'jobId': job_id,
            'result': result,
            'azureDevOpsResponse': str(response)
        })
    }