perlweb-scrapinglwp-useragent

Why does the HTML I fetch with Perl look different from what I see in browser?


I am writing a Web scraper using Perl to fetch data from the http://www.coupons.com/ The problem is that the HTML I fetch with LWP::UserAgent is different from what I see in the web browser. I am interested in the content of the JavaScript variable "CouponClubMember" and in HTML I receive with Perl this variable is empty.

Any ideas?


Solution

  • Using code below, I am getting same thing as with my browser. I just set agent to same string my Firefox sent and enabled cookie handling:

    use LWP::UserAgent;
    
    my $ua = LWP::UserAgent->new(
        agent      => 'Mozilla/5.0 (Windows NT 5.1; rv:7.0.1) Gecko/20100101 Firefox/7.0.1',
        cookie_jar => {},
    );
    
    $res = $ua->get("http://www.coupons.com");
    
    if($res->content =~ /(CouponClubMember.{300})/) {
        print $1;
    }