im trying this sample to obtain the number of offers a NFT has in opensea:
import { test, expect } from '@playwright/test';
test('test', async ({ page }) => {
await page.goto('https://opensea.io/assets/ethereum/0x63217dbb73e7a02c1d30f486e899ee66d0aa5e0b/6341');
await page.waitForLoadState('networkidle');
let selector = page.locator("[id='Body offers-panel'] li");
const offers = await selector.count();
console.log('Num of offers:', offers);
});
and then I run "npx playwright tests" what always print "Num of offers: 0"
But if I run it in --headed mode, it works perfectly and outputs "Num of offers: 5"
Can anyone explain/help me to understand it?
I tried using:
let selector = page.locator("[id='Body offers-panel'] li").waitFor();
Tried to wait until all requests are done
await page.waitForLoadState('networkidle');
tried to wait for the selector:
let selector = page.locator("[id='Body offers-panel'] li").first().waitFor();
But none worked, I always have 0 count unless I run the test in --headed mode, no matter of which NFT address I try.
I would like to solve it or understand why this happen
Headless mode makes it more obvious to servers that your script is a bot. You're being detected and blocked headlessly, but bypassing detection when running headfully.
Since you can't see anything, headless is a bit harder to debug than headful. The following
console.log(await page.content());
await page.screenshot({path: "test.png", fullPage: true});
are good tools for figuring out why elements you expect to be on the page aren't.
In this case, adding
const text = (await page.textContent("body"))
.replace(/ +/g, " ")
.replace(/(\n ?)+/g, "\n")
.trim();
console.log(text);
after goto
to get the full text content of the page gives:
Access denied
Error code 1020
You do not have access to <Your URL>.The site owner may have set restrictions that prevent you from accessing the site.
Error details
Provide the site owner this information.
I got an error when visiting <Your URL>.
Error code: 1020
Ray ID: **************
Country: US
Data center: *****
IP: *****************
Timestamp: 2023-02-17 22:39:13 UTC
Click to copy
Was this page helpful?
Yes
No
Thank you for your feedback!
Performance & security by Cloudflare
It's not a perfect guarantee, but adding a user agent header is an easy option that seems to be enough to avoid headless detection on this particular site at this point in time:
import {expect, test} from "@playwright/test"; // ^1.42.1
const url = "<Your URL>";
const userAgent =
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36";
test.describe("with user agent", () => {
test.use({userAgent});
test("is able to retrieve offers", async ({page}) => {
await page.goto(url, {waitUntil: "commit"});
const offers = page.locator('[id="Body offers-panel"] li');
await expect(async () => {
expect(await offers.count()).toBeGreaterThanOrEqual(10);
}).toPass();
});
});
This works because the default Playwright headless user agent header explicitly says "I am a robot" by default, while headful uses a normal browser user agent. But keep in mind bot detection involves many more factors than just this. Using a rotating proxy service would be a more robust solution.
For completeness, here's how to change the user agent in a non-test Playwright script, which is more typical in scraping:
const playwright = require("playwright"); // ^1.42.1
let browser;
let context;
(async () => {
const url = "<Your URL>";
const userAgent =
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36";
browser = await playwright.firefox.launch();
context = await browser.newContext({userAgent, bypassCSP: true});
const page = await context.newPage();
await page.goto(url, {waitUntil: "commit"});
const sel = '[id="Body offers-panel"] li';
await page.waitForFunction(
`document.querySelectorAll('${sel}').length >= 10`
);
const offers = await page.locator(sel).count();
console.log("Num of offers:", offers);
})()
.catch(err => console.error(err))
.finally(() => browser?.close());