python-3.xselenium-webdriverbeautifulsoupweb-crawlerquora

Unable to Extract JavaScript Elements while Scraping Quora


I am trying to extract Data from Quora for analysis purposes using Python, BeautifulSoup and Selenium. But I am unable to extract the JavaScript elements on the page. How should I extract them?

Here I am only trying to extract bio of the Quora Profile but I am not getting the text which comes after clicking the 'more' button.

~https://i.sstatic.net/20q72.jpg

                # Extracting Bio
                driver.find_element_by_class_name('ui_qtext_more_link').send_keys(Keys.ENTER)
                bio = driver.find_element_by_class_name("ui_qtext_rendered_qtext").text

Solution

  • Please use the below line of code to first click on the "more" button and then fetch the expanded text of the profile.

    import time
    //Fetch the more button element first
    WebElement moreButton = driver.find_element_by_xpath("(//a[@class='ui_qtext_more_link'])[1]");    
    //Click on the more button
    moreButton.click();
    time.sleep(3)
    //Fetch the profileInfo element
    WebElement profileInfo = driver.find_element_by_xpath("(//div[contains(@id,'expanded_content')]//span[@class='ui_qtext_rendered_qtext'])[1]");
    //Store the bio in a string and use it further
    String profileInfoBio = profileInfo.text;