I am trying to scrape a password protected website in python. My code is as follows:
import mechanize
import cookielib
from BeautifulSoup import BeautifulSoup
br = mechanize.Browser()
cj = cookielib.LWPCookieJar()
br.set_cookiejar(cj)
br.set_handle_equiv(True)
br.set_handle_gzip(True)
br.set_handle_redirect(True)
br.set_handle_referer(True)
br.set_handle_robots(False)
br.set_handle_refresh(mechanize._http.HTTPRefreshProcessor(), max_time=1)
br.addheaders = [('User-agent', 'Chrome')]
br.open('https://monitor1.returnpath.net/login.php')
for f in br.forms():
print f
br.select_form(nr=1)
br.form['email'] = 'email'
br.form['password'] = 'password'
The for loop returns this:
<form1 POST https://monitor1.returnpath.net/login.php application/x-www-form-urlencoded <TextControl(email=)> <PasswordControl(password=)> <CheckboxControl(remember=[1])> <SubmitControl(Submit=Sign In) (readonly)>> <GET http://now.eloqua.com/e/f2.aspx application/x-www-form-urlencoded <TextControl(e=)> <HiddenControl(lang=NA) (readonly)> <HiddenControl(elqSiteID=841) (readonly)> <HiddenControl(elqFormName=nLRegFooter-1347904420246) (readonly)> <SubmitControl(<None>=Sign Me Up) (readonly)> <SubmitButtonControl(<None>=) (readonly)>> <POST https://monitor1.returnpath.net/send_feedback.php application/x-www-form-urlencoded <HiddenControl(size=) (readonly)> <HiddenControl(nps=) (readonly)> <TextareaControl(desc=)> <IgnoreControl(<None>=<None>)> <IgnoreControl(<None>=<None>)> <IgnoreControl(<None>=<None>)> <IgnoreControl(<None>=<None>)> <IgnoreControl(<None>=<None>)> <IgnoreControl(<None>=<None>)> <IgnoreControl(<None>=<None>)> <IgnoreControl(<None>=<None>)> <IgnoreControl(<None>=<None>)> <IgnoreControl(<None>=<None>)> <IgnoreControl(<None>=<None>)>>
And this error:
mechanize._form.ControlNotFoundError: no control matching name 'email'
The output states that 'email' is found so I'm not sure why it says there is no control matching it?
Its zero-indexed. try the code below:
br.select_form(nr=0)