mechanicalsoup

Logging into Yahoo Finance Using MechanicalSoup


Right now I have been trying to access yahoo with python and am I am not sure why I can't seem to login.

My intended flow is

go to yahoo -> go to login -> enter username -> press submit button -> enter password -> press submit button.

Please let me know where I have made a mistake and why not code doesn't seem to work. Any alternatives to login into yahoo that are not selenium-based would be appreciated and still use python.

"""Example app to login to Yahoo using the StatefulBrowser class."""

from __future__ import print_function
import argparse
import mechanicalsoup

browser = mechanicalsoup.StatefulBrowser(
    soup_config={'features': 'lxml'},
    raise_on_404=True,
    user_agent='MyBot/0.1: mysite.example.com/bot_info',
)
# Uncomment for a more verbose output:
browser.set_verbose(2)
browser.session.cookies.keys()
browser.open("https://login.yahoo.com/config/login?.src=fpctx&.intl=ca&.lang=en-CA&.done=https%3A%2F%2Fca.yahoo.com")

form1 = browser.select_form(nr=0)
browser['username'] = 'beta@gmail.com'
response = browser.submit_selected()
print(response.content)
browser.select_form(nr=0)
browser['passwd'] = 'badPass'
response = browser.submit_selected()
print(response)
page = browser.get_current_page()

Solution

  • A quick look at the login page source shows that it uses JavaScript quite extensively. It seems very likely that the form submission is handled by JavaScript, though I can't point to an exact line of code that proves this incontrovertibly.

    Since MechanicalSoup does not support JavaScript, you may need to find an alternate tool that does, such as Selenium. See this FAQ for more information.