google-apps-scriptweb-scrapinghttp-status-code-429

Getting data from another url in Appscript

I'm trying to scrape data from the site. The request url that get the product info shows in browser properly. But not getting the UrlFetchApp. My code is as following

function myFunction() {  
  var coles_url = "https://shop.coles.com.au/search/resources/store/20520/productview/bySeoUrlKeyword/mutti-tomato-passata-2349503p?catalogId=12064";
  Logger.log(jsonInitColes(coles_url));
  
}

function jsonInitColes(url){
   var options =
  {
   "method"  : "GET",   
   "followRedirects" : true,
   "muteHttpExceptions": true
    };
   var response = UrlFetchApp.fetch(url,options);
   var content = response.getContentText();
   return content; 
}

But I was getting the following HTML instead of data status_code is 429(Too many requests)

 <!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <link rel="shortcut icon" href="about:blank">
</head>
<body>
<script src="/149e9513-01fa-4fb0-aad4-566afd725d1b/2d206a39-8ed7-437e-a3be-862e0f06eea3/j.js"></script>
<script src="/149e9513-01fa-4fb0-aad4-566afd725d1b/2d206a39-8ed7-437e-a3be-862e0f06eea3/f.js"></script>
<script src="/149e9513-01fa-4fb0-aad4-566afd725d1b/2d206a39-8ed7-437e-a3be-862e0f06eea3/fingerprint/script/kpf.js?url=/149e9513-01fa-4fb0-aad4-566afd725d1b/2d206a39-8ed7-437e-a3be-862e0f06eea3/fingerprint&token=c3436f36-e3c3-a537-25f3-bedc8592f189"></script>
</body>
</html>

Browser inspection of the request enter image description here

enter image description here

Solution

  • Answer:

    There's nothing you can do about this.

    More Information:

    Google Apps Script runs in the cloud, so your fetch will be coming from some unspecified Google IP.

    Imagine how many users are using Google Apps Script. Of those users, how many are trying to fetch information from the same sites and servers. The Too many Requests is an unfortunately common error when using UrlFetchApp to external endpoints that don't require authentication

    If you're getting a Too many requests error and you're not doing quick retries, then it's likely that there are just too many requests coming from all those other Apps Script scripts in the cloud making requests to the same server. There's nothing you can do about that.