pythonweb-scrapingxmlhttprequestmechanize

How to simulate a AJAX call (XHR) with python and mechanize


I am working on a project that does online homework automatically. I am able to login, finding exercises and even filling the form using mechanize. I discovered that the submit button trigger a javascript function and I searched for the solution. A lot of answers consist of 'simulating the XHR'. But none of them talked about the details. I don't know if this screen cap helps. https://i.sstatic.net/0g83g.png Thanks


Solution

  • If you want to evaluate javascript, I'd recommend using Selenium. It will open a browser which you can then send text to it from python.

    First, install Selenium: https://pypi.python.org/pypi/selenium

    Then download the chrome driver from here: https://code.google.com/p/chromedriver/downloads/list

    Put the binary in the same folder as the python script you're writing. (Or add it to the path or whatever, more information here: https://code.google.com/p/selenium/wiki/ChromeDriver)

    Afterwards the following example should work:

    from selenium import webdriver
    from selenium.webdriver.common.keys import Keys
    
    driver = webdriver.Chrome()
    driver.get("http://www.python.org")
    assert "Python" in driver.title
    elem = driver.find_element_by_name("q")
    elem.send_keys("selenium")
    elem.send_keys(Keys.RETURN)
    assert "Google" in driver.title
    driver.close()
    

    More information here (The example was also from there)