AWS API Gateway / AWS Network Load Balancer: request ends in 500 InternalServerErrorException, How to specify the port to forward to?

I have been struggling with this problem for 2 days but couldn't get it working.

I have this flow:

external world --> AWS API Gateway ---> VPC Link ---> Network Load Balancer ---> my single EC2 instance

I have set up the EC2 instance correctly. A "Typescript/ExpressJS" api service is running on port 3001

I have also set up a Network Load Balancer and A Target Group, the NLB is listening and forwarding requests to port 3001 of the target group (which contains the EC2 instance).

I have configured the security group of my EC2 instance such that:

1. Allow All protocol traffic on All ports from my VPC
 (specified using CIDR notation `171.23.0.0/16`);

2. Allow All protocol traffic on All ports from ANY source (`0.0.0.0/0`);

Now, when I do curl docloud-backend-xxxxx.elb.ap-northeast-1.amazonaws.com:3001/api/user,

the api service gets the request and I can see logs generated in the EC2 instance.

Then, I have also set up my API Gateway, but the problem seems to be that requests made to the API gateway are not delivered to port 3001 of my EC2 instance.

In the API Gateway configurations, when I set up VPC Link and specify the Target NLB, it looks like the API Gateway will send requests to port 80 by default.

I have to tell the API Gateway to send requests to docloud-backend-xxxxx.elb.ap-northeast-1.amazonaws.com:3001.

However, there is no place to set the "forward-to" port number on AWS API Gateway.

How can I configure?

Below is the detailed setting: On the AWS API Gateway side, I have:

1. set up a "VPC Link" that points to my VPC and configured the "Target NLB" to be my NLB:

2. I have configured custom domain:

and I have set up the API mapping:

3. On the resources of the REST API, I have configured a proxy resource:

I have completed all the necessary settings that I know of from the AWS documentations, but now the problem is that requests made to the API gateway do not seem to be delivered to my EC2 instance.

On the API Gateway's Method Test, I get the following error:

Request: /api/user
Status: 500
Latency: 4282 ms
Response Body
{
  "message": "Internal server error"
}
Response Headers
{"x-amzn-ErrorType":"InternalServerErrorException"}
Logs
Execution log for request 22ac2b9b-e859-474d-aa69-bd8c4bf51b4b
Tue Dec 28 09:53:02 UTC 2021 : Starting execution for request: 22ac2b9b-e859-474d-aa69-bd8c4bf51b4b
Tue Dec 28 09:53:02 UTC 2021 : HTTP Method: GET, Resource Path: /api/user
Tue Dec 28 09:53:02 UTC 2021 : Method request path: {proxy=api/user}
Tue Dec 28 09:53:02 UTC 2021 : Method request query string: {}
Tue Dec 28 09:53:02 UTC 2021 : Method request headers: {}
Tue Dec 28 09:53:02 UTC 2021 : Method request body before transformations: 
Tue Dec 28 09:53:02 UTC 2021 : Endpoint request URI: http://docloud-backend-6ff16c1d637f19bc.elb.ap-northeast-1.amazonaws.com
Tue Dec 28 09:53:02 UTC 2021 : Endpoint request headers: {x-amzn-apigateway-api-id=jeva09t4qc, User-Agent=AmazonAPIGateway_jeva09t4qc, Host=docloud-backend-6ff16c1d637f19bc.elb.ap-northeast-1.amazonaws.com}
Tue Dec 28 09:53:02 UTC 2021 : Endpoint request body after transformations: 
Tue Dec 28 09:53:02 UTC 2021 : Sending request to http://docloud-backend-6ff16c1d637f19bc.elb.ap-northeast-1.amazonaws.com
Tue Dec 28 09:53:07 UTC 2021 : Execution failed due to configuration error: There was an internal error while executing your request
Tue Dec 28 09:53:07 UTC 2021 : Method completed with status: 500

From the browser's dev mode, I can see 2 requests made to endpoint /api/user. One is a preflight request which ends with 200 success, the other is the "actual" request that gives 500 and InternalServerErrorException error:

I think the requests has mode its way from API Gateway to the NLB, but it hits port 80 instead of 3001.

How can I set it right?

After configuring and reading the flow logs for this NLB’s network interface, it is clear that API Gateway is not sending any traffic into the NLB; This is how I fix the problem as of 2024-12:

• create a new private subnet
• create a new NLB placed in the private subnet
• create a new VPC link in API Gateway console
• update the API Gateway API with the new NLB’s domain name
• make sure security group work for NLB and EC2 instance;