2 years back Google introudced private DNS which is very important for internal domains communication.

I setup a private dns to learn about DNS using below documentation guide https://cloud.google.com/sdk/gcloud/reference/dns/managed-zones/create https://www.jhanley.com/google-cloud-private-dns-zones/

Below are the steps followed to create an example zone

Created a private zone "private-zone"

gcloud dns managed-zones create --dns-name="example.com" --description="Private Zone" --visibility=private --networks=default "private-zone"

Then created a vm in google cloud and ran the nslookup for the domain name. But it did not resolve

testdns:~$ nslookup example.com
Server:         169.254.169.254
Address:        169.254.169.254#53
Non-authoritative answer:
*** Can't find example.com: No answer

I'm using Debian OS on the Vm instance created

Linux testdns 4.19.0-12-cloud-amd64 #1 SMP Debian 4.19.152-1 (2020-10-18) x86_64 GNU/Linux

These are two A records other than NS and SOA records in

test.example.com.   A   300  192.0.0.9
www.example.com.    A   300  192.0.0.91
example.com.        A   3600 192.0.1.1

Below example was tested

nslookup when "example.com" A record was not added

mymach@testdns:~$ nslookup example.com ns-gcp-private.googledomains.com
Server:         ns-gcp-private.googledomains.com
Address:        169.254.169.254#53
Non-authoritative answer:
*** Can't find example.com: No answer

#dig example.com using nameserver , 'A' record was added

testdns:~$ dig example.com @ns-gcp-private.googledomains.com
; <<>> DiG 9.11.5-P4-5.1+deb10u2-Debian <<>> example.com @ns-gcp-private.googledomains.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41534
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;example.com.                   IN      A
;; ANSWER SECTION:
example.com.            3600    IN      A       192.0.1.1
;; Query time: 12 msec
;; SERVER: 169.254.169.254#53(169.254.169.254)
;; WHEN: Tue Dec 08 23:03:58 UTC 2020
;; MSG SIZE  rcvd: 56

nslookup again after adding example.com

testdns:~$ nslookup example.com ns-gcp-private.googledomains.com
Server:         ns-gcp-private.googledomains.com
Address:        169.254.169.254#53
Non-authoritative answer:
Name:   example.com
Address: 192.0.1.1

dig

testdns:~$ dig example.com 
; <<>> DiG 9.11.5-P4-5.1+deb10u2-Debian <<>> example.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 24673
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;example.com.                   IN      A
;; ANSWER SECTION:
example.com.            3600    IN      A       192.0.1.1
;; Query time: 11 msec
;; SERVER: 169.254.169.254#53(169.254.169.254)
;; WHEN: Tue Dec 08 23:06:30 UTC 2020
;; MSG SIZE  rcvd: 56

dig using local host

dig example.com @127.0.0.1
; <<>> DiG 9.11.5-P4-5.1+deb10u2-Debian <<>> example.com @127.0.0.1
;; global options: +cmd
;; connection timed out; no servers could be reached

Am I missing some step ?

After you created your zone with:

gcloud dns managed-zones create --dns-name="example.com" --description="Private Zone" --visibility=private --networks=default "private-zone"

You must create DNS registries to it, for example:

gcloud dns record-sets transaction start --zone="private-zone"
gcloud dns record-sets transaction add 10.2.3.4 --name="example.com" --ttl="3600" --type="A" --zone="private-zone"
gcloud dns record-sets transaction execute --zone="private-zone"

Give GCP 1 minute to catch up, and then try again with the default "/etc/resolv.conf" file.