TL;DR SOLUTION: DIG CANNOT BE USED TO VERIFY DNS FORWARDING POLICIES AS THE DNS RESOLUTION IS PERFORMED BY THE ROOT SERVERS
I am trying to create a DNS outbound forwarding policy for forwarding DNS requests to a specific name server. It is going to be implemented in an on-prem setup, but I am currently just testing it with public DNS servers such as 1.1.1.1. I have created the DNS outbound policy, but every request from inside VMs still uses the metadata server: 169.254.169.254
CURRENT SETUP: I have created a VPC, defined a subnet, and created a VM connected to this subnet.
resource "google_compute_network" "vpc-hub" {
name = "vpc-hub"
depends_on = [
google_project_iam_member.assign-roles,
google_project_service.enable_apis
]
}
resource "google_compute_subnetwork" "vpc-subnet-hub" {
name = "vpc-subnet-hub"
ip_cidr_range = "10.0.0.0/16"
region = "europe-west1"
network = google_compute_network.vpc-hub.id
depends_on = [
google_compute_network.vpc-hub
]
}
resource "google_compute_instance" "hub-vm" {
name = "hub-vm"
machine_type = "e2-medium"
zone = "europe-west1-b"
depends_on = [
google_compute_subnetwork.vpc-subnet-hub
]
boot_disk {
initialize_params {
image = "debian-cloud/debian-9"
}
}
network_interface {
subnetwork = google_compute_subnetwork.vpc-subnet-hub.name
access_config {
}
}
metadata_startup_script = file("${path.module}/startup_install.sh")
}
The startup-script installs dnsutils. Additionally, I have defined a firewall rule to be able to connect to it via SSH:
resource "google_compute_firewall" "ssh-rule-hub" {
name = "demo-ssh"
network = google_compute_network.vpc-hub.name
allow {
protocol = "tcp"
ports = ["22"]
}
source_ranges = ["REDACTED]
depends_on = [
google_compute_network.vpc-hub
]
}
ISSUE: I have created a DNS forwarding policy by specifying the public name server 1.1.1.1:
resource "google_dns_policy" "dns-policy" {
name = "dns-policy"
enable_inbound_forwarding = false
enable_logging = false
alternative_name_server_config {
target_name_servers {
ipv4_address = "1.1.1.1"
forwarding_path = "default"
}
}
networks {
network_url = google_compute_network.vpc-hub.id
}
depends_on = [
google_compute_network.vpc-hub
]
}
When I SSH into the VM and attempt to use dig to send a DNS request to any domain. It always uses the internal metadata server within the compute engine - And thus does not use 1.1.1.1 to retrieve the IP. Instead the answer is always provided by the metadata server: 169.254.169.254. Why is this the case? Output from Dig can be seen below:
bash command: dig google.com
; <<>> DiG 9.10.3-P4-Debian <<>> google.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 60323
;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;google.com. IN A
;; ANSWER SECTION:
google.com. 300 IN A 64.233.184.101
google.com. 300 IN A 64.233.184.139
google.com. 300 IN A 64.233.184.102
google.com. 300 IN A 64.233.184.100
google.com. 300 IN A 64.233.184.113
google.com. 300 IN A 64.233.184.138
;; Query time: 18 msec
;; SERVER: 169.254.169.254#53(169.254.169.254)
;; WHEN: Tue Apr 26 12:09:53 UTC 2022
;; MSG SIZE rcvd: 135
EDIT: output from bash command dig google.com +trace
; <<>> DiG 9.10.3-P4-Debian <<>> google.com +trace
;; global options: +cmd
. 510859 IN NS a.root-servers.net.
. 510859 IN NS b.root-servers.net.
. 510859 IN NS c.root-servers.net.
. 510859 IN NS d.root-servers.net.
. 510859 IN NS e.root-servers.net.
. 510859 IN NS f.root-servers.net.
. 510859 IN NS g.root-servers.net.
. 510859 IN NS h.root-servers.net.
. 510859 IN NS i.root-servers.net.
. 510859 IN NS j.root-servers.net.
. 510859 IN NS k.root-servers.net.
. 510859 IN NS l.root-servers.net.
. 510859 IN NS m.root-servers.net.
;; Received 239 bytes from 169.254.169.254#53(169.254.169.254) in 22 ms
com. 172800 IN NS k.gtld-servers.net.
com. 172800 IN NS b.gtld-servers.net.
com. 172800 IN NS l.gtld-servers.net.
com. 172800 IN NS i.gtld-servers.net.
com. 172800 IN NS e.gtld-servers.net.
com. 172800 IN NS j.gtld-servers.net.
com. 172800 IN NS a.gtld-servers.net.
com. 172800 IN NS f.gtld-servers.net.
com. 172800 IN NS g.gtld-servers.net.
com. 172800 IN NS d.gtld-servers.net.
com. 172800 IN NS c.gtld-servers.net.
com. 172800 IN NS m.gtld-servers.net.
com. 172800 IN NS h.gtld-servers.net.
com. 86400 IN DS 30909 8 2 E2D3C916F6DEEAC73294E8268FB5885044A833FC5459588F4A9184CF C41A5766
com. 86400 IN RRSIG DS 8 1 86400 20220510050000 20220427040000 47671 . hGerEs3N471ZCOosNSuakxBfxXh8H+qPP9UxVBakXvVfgLofu40+aNyw X9tfaNxmyFP7LUJDRBrURhNjN1cOdJhbTqa54AXvTlPbd31N5MRF3ZHT seJJLCe8Hv2UYLrnLSzAArpD2M+N0XI+3A6wR8/fE4/q0NULX0gpKxS9 Y/zHpr/Mu/2I8DLmI8sE411vSlK2MFxWj2LfQ0TAocjnmqkZQK9GfqQ4 IEDjcc2OV41JlxSKYtAy9OI3HJPfXcIrmo4aO9Qvoe1ZKn1fu46IUYpo zx1n5NgX4Ou8kOgjePMkpGMlvjLVY2KMxYdln7v4FCRB7mb1WsBp5H+3 3Sv7vQ==
;; Received 1170 bytes from 192.33.4.12#53(c.root-servers.net) in 10 ms
google.com. 172800 IN NS ns2.google.com.
google.com. 172800 IN NS ns1.google.com.
google.com. 172800 IN NS ns3.google.com.
google.com. 172800 IN NS ns4.google.com.
CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN NSEC3 1 1 0 - CK0Q1GIN43N1ARRC9OSM6QPQR81H5M9A NS SOA RRSIG DNSKEY NSEC3PARAM
CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN RRSIG NSEC3 8 2 86400 20220503042354 20220426031354 37269 com. vPwsL+I4ptKlrtrsZXrs/Z3/1Ebd3eBCm5f73FcuoljLz9RHDPdxfGCd rHmK3oWXZIErmCLYy6NgcpiMEM+JiqTQm/mNFhVfoFTGicCB9JwGgA3p XfgAB6w/hsNvtqTVuFsszhq4+CjXf1+ky0hFzTHACJ3bPmEeMa80ASwX EHu+vpL9LdLSZrtS68G2ieGvjrwcn6EZfmfJIly3i8UU9g==
S84BKCIBC38P58340AKVNFN5KR9O59QC.com. 86400 IN NSEC3 1 1 0 - S84BUO64GQCVN69RJFUO6LVC7FSLUNJ5 NS DS RRSIG
S84BKCIBC38P58340AKVNFN5KR9O59QC.com. 86400 IN RRSIG NSEC3 8 2 86400 20220504051944 20220427040944 37269 com. G/tnTZRrBZApHzz+bWDsUrDBa6PcH5O15adPaWvIlvIMe1FQUYFRJeMm pzJ586JzBfz5bE+5IjU0uAh6AnWrDVtryUig0CUpMVDn8mm8Axa9QTu/ 1h3hwE6HbKx/4d7hNpZIuKGH98iMKJLaYdfJHFLRV85/AOT9HdL2ihtF t2wOZ3Px0Wo6QVxpwth2hO2iLw0k/tszErnHw6BqxEAebQ==
;; Received 836 bytes from 192.41.162.30#53(l.gtld-servers.net) in 8 ms
google.com. 300 IN A 142.251.5.101
google.com. 300 IN A 142.251.5.113
google.com. 300 IN A 142.251.5.100
google.com. 300 IN A 142.251.5.138
google.com. 300 IN A 142.251.5.102
google.com. 300 IN A 142.251.5.139
;; Received 135 bytes from 216.239.32.10#53(ns1.google.com) in 1 ms
The SERVER address 169.254.169.254 does not mean the server that responded. That address is the metadata server that knows how to forward internal (project specify) and global domain names to the correct DNS server. That address is a virtualized service that provides a number of services such as DNS, DHCP, and NTP.