[SOLVED] Xpath: Working in Devtools but returns empty object with document.evaluate()

Xpath: Working in Devtools but returns empty object with document.evaluate()

The following command works as expected in Devtools console (for example, here - https://news.ycombinator.com/)

$x('//a[@class="storylink"]') (Edge browser)

But the following code:

            const page = await browser.newPage();
            await page.goto("https://news.ycombinator.com/");
            let urls = await page.evaluate(() => {
                var item = document.evaluate(
                    '//a[@class="storylink"]',
                    document,
                    null,
                    XPathResult.FIRST_ORDERED_NODE_TYPE,
                    null
                ).singleNodeValue;
                return item
            })
            browser.close();

returns an empty object: {}. The same is happening in every other website. Why is that happening?

Solution

If you are automatic Chrome with Puppeteer from Node.js then the page object you have already exposes a method $x for XPath evaluation, see https://pptr.dev/#?product=Puppeteer&version=v10.4.0&show=api-pagexexpression. That means that doing

let urls = await page.$x('//a[@class="storylink"]')

should suffice.