javascriptpuppeteer

Unable to click on <a> tag using Puppeteer

I am trying to scrape a real estate website. This is my first time using Puppeteer. Up to now I have managed to open the link and click on the "accept all" button for cookies. Now I am trying to click on the "sign in" button. This is my code:

const puppeteer = require("puppeteer");

const link1 =
  "https://www.daft.ie/for-rent/walled-garden-wyckham-way-dundrum-dublin-14/3988400";

(async () => {
  const browser = await puppeteer.launch({
    headless: false,
    defaultViewport: null,
  });
  const page = await browser.newPage();
  await page.goto(link1);

  const elements = await page.$x("//button[normalize-space()='Accept All']");
  await elements[0].click();
  console.log("cookies accepted");

  const signin = await page.$x("//a[contains(., 'Sign in')]");
  await signin[0].click();

  console.log("Sign in clicked");

  await browser.close();
})();

This is the error I get after I run the script:

 throw new Error('Execution context was destroyed, most likely because of a navigation.');
              ^

Error: Execution context was destroyed, most likely because of a navigation.
    at rewriteError (/Users/devang/Desktop/coding/daft/node_modules/puppeteer/lib/cjs/puppeteer/common/ExecutionContext.js:280:15)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async ExecutionContext._ExecutionContext_evaluate (/Users/devang/Desktop/coding/daft/node_modules/puppeteer/lib/cjs/puppeteer/common/ExecutionContext.js:226:56)
    at async ElementHandle.evaluateHandle (/Users/devang/Desktop/coding/daft/node_modules/puppeteer/lib/cjs/puppeteer/common/JSHandle.js:97:16)
    at async Object.internalHandler.queryAll (/Users/devang/Desktop/coding/daft/node_modules/puppeteer/lib/cjs/puppeteer/common/QueryHandler.js:63:30)
    at async ElementHandle.$$ (/Users/devang/Desktop/coding/daft/node_modules/puppeteer/lib/cjs/puppeteer/common/ElementHandle.js:116:17)
    at async /Users/alex/Desktop/coding/daft/index.js:18:1

I verified the xPath I am using for the a tag sign in button is correct.

Here's the HTML of the sign in link:

<a data-testid="top-level-active-nav-link" href="/auth/authenticate" class="Navigationstyled__TopLevelItemActiveLink-sc-1tagpb7-6 bsFoce">Sign in</a>

What am I doing wrong here?

Link of the target page - list

Solution

  • page.click() issues a trusted event that sometimes fails due to visibility issues. A workaround is using element.evaluate(el => el.click()) to trigger a native DOM click.

    import puppeteer from "puppeteer"; // ^23.10.4

let browser;
(async () => {
  browser = await puppeteer.launch({headless: false});
  const [page] = await browser.pages();
  const url =
    "https://www.daft.ie/for-rent/walled-garden-wyckham-way-dundrum-dublin-14/3988400";
  await page.goto(url, {waitUntil: "domcontentloaded"});
  const signIn = await page.waitForSelector("::-p-text(Sign in)");
  await signIn.evaluate(el => el.click()),
  await page.waitForSelector("#username");
  await page.type("#username", "hello");
  await page.type("#password", "world");
  await page.click("#login");

  // just to show it's working
  await page.waitForSelector("::-p-text(Invalid username)");
  await page.screenshot({path: "proof.png"});
})()
  .catch(err => console.error(err))
  .finally(() => browser?.close());

    You can typically skip dismissing the cookie banner, or call .remove() on its parent container to rip it out entirely if it's in the way.

    Going a step further, it's often a good idea to navigate directly to the sign in page URL, skipping a button click and navigation that could fail.

    Another trick I use for scripts with complex sign-in workflows that involve a lot of frames, oauth and other difficult-to-automate components is to do it all by hand in Puppeteer's browser while saving the cookies and site data in the userDataDir. It seems that this website should be workable without that trick but it's worth mentioning in case you run into trouble.