google-chromecookiespuppeteergdprconsentform

What are the necessary cookies to set so the "Before you continue to Google" page won't appear?


I'm scraping data from Google using Puppeteer. But before Puppeteer gets to the google page, an annoying popup appears (screenshot).

I want to prevent this from happening so that I don't have to click the "reject" / "allow" button in Puppeteer every time. What are the necessary cookies to achieve it?


  const browser = await puppeteer.launch()
  const page = await browser.newPage()
  await page.goto("https://google.com/")

      const cookies = [
    {
      domain: '.google.com',
      expirationDate: 9999999999.298648,
      name: 'CONSENT',
      ...
    },
  ]

  await page.setCookie(...cookies)

Solution

  • Instead of trying to hard code the cookies, just save them.

    const cookiesPath = './cookies.json';
    const browser = await puppeteer.launch()
    const page = await browser.newPage()
    
    if(await fs.existsSync(cookiesPath)){
      let cookies = require(cookiesPath)
      for(let cookie of cookies){
        await page.setCookie(cookie)
      }
    }
    
    await page.goto("https://google.com/")
    
    const hasConsentForm = async ()=>{
      // create query that will return consent form
      const consentFormQuerySelector=''
      // I dont remember if page.$eval querySelector fails 
      // throws errors or not
      try{
        return await page.$eval(consentFormQuerySelector,el=>Boolean(el))
      }catch(err){
        console.log(err)
        return false
      }
    }
    
    const navigateConsentForm = async ()=>{
      // add your existing code that navigates the consent form
      
      // ...
    
      // after the consent form is cleared, save cookies
      let cookies = await page.cookies()
      fs.writeFileSync(cookiesPath,JSON.stringify(cookies,null,2))
    }
    
    if(await hasConsentForm()){
     await navigateConsentForm()
    }
    

    Assuming that the lack of cookies or the lack of a certain cookie property is the reason that the consent form pops up, this is will make it so that after running once, the form doesnt show again.