phpweb-scrapingselenium-chromedriverwebchromeclientsymfony-panther

Symfony/panther web Scraping not working for content behind login (Cloud function)

use \Symfony\Component\Panther\Client;
require __DIR__ . '/vendor/autoload.php';

$options = [
    '--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36',
    '--disable-blink-features=AutomationControlled',
    '--enable-javascript'
];

$client = Client::createChromeClient(null, null, $options);
$url = 'https://example.ai/profile/';
$client->request('GET', $url);

I am trying to create a simple web scraper script for personal interest (learning) with Symfony/panther. The target web application most probably hosted on Google Firebase and also using it's authentication system. Once my script clicking on the Login button, it successfully taking to the profile page. But it's meaningless because when I try to navigate to other page it's again functioning as anonymous user. It's not holding the authentication.

I have checked from real browser, On clicking on the login button It's sending POST request to the following url endpoints:

identitytoolkit.googleapis.com/v1/accounts:signInWithPassword?key=somekey

identitytoolkit.googleapis.com/v1/accounts:lookup?key=somekey

Also some additional post request like:

us-central1-dreampen-xxxxx.cloudfunctions.net/getSubscriberInfo

Here is the error I am getting: enter image description here

If anyone have any idea about this issue please show some tips:(

Solution

  • Never Mind! I have fixed the problem.