DNS Record for EC2 in private subnet accessible via Client VPN

We have an EC2 in our private subnet hosting a Jenkins server. We have a Client VPN Endpoint to connect to this private subnet. Our EC2 SG allows Inbound HTTP requests from the VPN SG and this VPN SG allows Outbound HTTP requests to our EC2 SG. So while connected to the VPN, we can connect to the EC2 using its Private IP address.

We now want to add a DNS Record on top of this Private IP address.

We already have a private Hosted Zone associated with the right VPN. Our Endpoint VPN references the 2 DNS Server IP Addresses meaning that we've filled the DNS servers field of the Client VPN Endpoint with the two IP Addresses of our Route 53 Resolver Inbound Endpoint. "DNS resolution" and "DNS hostnames" are enabled in our VPC. Finally, we have a simple A Record whose value is the EC2 Private IP.

When logged into our VPN, the DNS Record is still not working. What are we missing?

Complementary information:

• Reaching the private EC2 through its DNS while logged in another EC2 in the same private subnet works.
• We have 2 VPC on this Private Hosted Zone, each VPC has 2 Availability Zones each one having one Resolver Inbound Endpoint IP Address. I've found this troubleshooting doc that states, in the section Confirm that the Resolver rule and its inbound endpoint resolve to different VPCs that there can be an infinite loop.
• Our VPN allows to enter only 1 VPC, which is the good one.

Queries ran on my laptop that could help

❯ dig my.dns.internal

;; communications error to 127.0.0.53#53: timed out

; <<>> DiG 9.18.12-0ubuntu0.22.04.3-Ubuntu <<>> my.dns.internal
;; global options: +cmd
;; no servers could be reached

❯ cat /etc/resolv.conf

nameserver 127.0.0.53
options edns0 trust-ad
search .

❯ resolvectl status
...
Link 4 (tun0)
    Current Scopes: DNS
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: resolverIp1
       DNS Servers: resolverIp1 resolverIp2

Doc checked on this topic:

• A re:Post: How do I resolve resource records in my private hosted zone using Client VPN?
• Another one Client VPN Connection to Route 53 Private Hosted Zone
• AWS doc on Alias Record

Quick solution

The issue was that my Resolver Inbound Endpoint IP Addresses, meaning its Network Interface's Security Group, were closed to my VPN. To fix this you have to:

• Add in the Inbound Rules of the security group of your Resolver Network Interface the rule: Type = DNS (TCP), Port: 53, Source: VPN Security Group
• Add in the Outbound rules of the security group of your VPN the rule: Type = DNS (TCP), Port: 53, Source: security group of your Resolver Network Interface

If it still doesn't work

To be checked

• A Private Hosted Zone associated with the right VPC and it contains your A Record pointing to the Private IP of your EC2
• The private subnet of your EC2 is in the right VPC
• The EC2 SG should be opened in Inbound Rule to HTTP (or any other you need) from your VPN
• The VPN SG should be opened in Outbound Rule to HTTP from your EC2, and to DNS (TCP) from your Resolver SG
• The Resolver SG should be opened in Inbound Rule to DNS (TCP) from your VPN
• Your Endpoint VPN references the 2 DNS Server IP Addresses meaning that you've filled the DNS servers field of the Client VPN Endpoint with the two IP Addresses of your Route 53 Resolver Inbound Endpoint
• "DNS resolution" and "DNS hostnames" are enabled in our VPC

Debug tools

• Check you can access your EC2 is reachable from its private IP, for example with: curl http://myPrivateIP, you can check this from your VPN and if it's not working check it from a private EC2 in the same subnet
• Check your private EC2 is reachable from another EC2 in the same private subnet, for example with: curl http://mydns
• Check your DNS Resolver is visible by your computer using resolvectl status
• Check your DNS using dig my.dns.internal