I am trying to scrape the new.google.com and get some information, I have no issue locally but when I deploy to our Datacenter it fails with "connection refused" that means it is blocked.
Get https://news.google.com?ceid=en%3Agb&gl=en-gb&hl=en-gb&hs=en-gb&pz=1: dial tcp 172.217.5.206:443: connect: connection refused
Do we have any alternative of passing header and by-pass to get unblocked? or is using paid API is the only option if I have to use google for some of my testing?
Did anyone encounter and resolved it?
This is pretty common. Somebody else used the same IP address for scraping or even something worse :) so it's blocked.
You can use some proxy services. There are some with free tier so it will do the job for testing.
And before you ask... free proxy services are super slow and probably already blocked :)