I am trying to use Google Data Fusion to connect to a Microsoft SQL server database and need to have a static IP.
I have tried to provision a static IP on a subnet and connect it to Data Fusion through a VPC and I have also created a small VM with a static IP then put it on the same VPC as the Cloud data fusion instance thinking I can connect the two through that but that was not successful.
I have limited experience with networking in Google Cloud Platform (GCP) and I'm seeking guidance on how to connect either Data Fusion or Cloud Composer to a static IP address. This is necessary for me to perform data ingestion from an external database.
This is a common problem I would imagine and whilst I have found similar questions, non seem to have clear easy to follow answers. If possible one with screen shots in the UI would be great too to try and understand further.
Relates to:
Indeed, if you have a limited experience on Google Cloud or Networking, it's not obvious at the first glance.
Firstly, you need to have in mind that Data fusion runs a job on a Dataproc cluster. And Composer runs on a GKE Autopilot cluster.
The similarity is: it's a cluster! Because of it, there are several VMs involved and all can't have a public IP (because of exhaustion of them). So all the VM have a private IP and not a public one.
To be able to reach the internet, you need a public IP. That's why you need to create a bridge that remember the requester (the VM and its private IP) and map the request to a public IP. This mechanism is named Network Address Translation, or NAT for short.
On Google Cloud, you can use Cloud NAT to perform it: you create your Cloud NAT, select the subnet that you want to NAT and that's all! More detail here
If you need to grant a public IP on your MySQL server, you can reserve a public IP and put it in the Cloud NAT configuration