I have Google Cloud Task queue deployed in us-east4 and basic http task handler deployed in GKE. The handler service is exposed via Internal application load balancer.
I can curl(http) to the IP of internal lb and get response from handler. However, when I create task with the same url and push them the task queue, handler never receives then and they live in the queue until the retry count is exhausted.
Note: Both Internal LB and cloud task queue are deployed in the same location us-east4.
When I convert the LB to be external, the connectivity works, however it seems counterintuitive to route the traffic via external LB.
Deploying the handler in App Engine is possible, but I wanted to explore the GKE approach as all other supporting services are also deployed there.
Any pointers are highly appreciated.
It's not possible for now. In fact, your internal load balancer exposes an IP in your own VPC.
Cloud Task is a serverless product and it is deployed in a Google Cloud VPC.
That means, Cloud Task is not connected to your VPC and doesn't know (reach) your internal load balancer.
It's the same with all the serverless product. However, some offers VPC connector (serverless VPC connector or direct VPC access for Cloud Run), but this option is not possible with Cloud Task (and PubSub, and Cloud Workflow, and Cloud Scheduler,...)
That being said, you have 2 solutions:
Note: you can set Cloud Functions and Cloud Run "ingress internal" (to prevent any external access), Cloud Task is compliant with "ingress internal" feature