pythonweb-scrapinghelium

Python helium get contents of table after click


I am using helium to scrape a webpage.

result = start_firefox(                                                                                   
    "https://www.medtronic.com/covidien/en-us/products/brain-monitoring/bis-monitoring-system.html",      
)                                                                                                         
                                                                                                          
result.find_element_by_class_name("js-open-table-overlay").click()   

After the click action i am presented with a table and i need to scrape the contents of the table but How do i select the table after the click ?


Solution

  • You would need to use find_elements_... to get all <table>, and use for-loop to work with every table separatelly amd use (nested) for-loop to get <tr> (row) and <th> (header) in table, and use (nested) for-loop to get <td> (cell) in row. And you would have to add elements to correct (nested) lists.


    But this page uses normal <table> and this table is all time in HTML (not added by javaScript) so it can be simpler to use pandas.read_html() to get all <table>

    import pandas as pd
    
    url = "https://www.medtronic.com/covidien/en-us/products/brain-monitoring/bis-monitoring-system.html"
    
    all_tables = pd.read_html(url)
    
    for table in all_tables:
        print(table.to_string())
    

    Result

         ORDER CODE                                                      DESCRIPTION UNIT OF MEASURE  QUANTITY
    0      186-1014                        BIS™ Complete 4-Channel Monitoring System            Each         1
    1      186-0210                                  BIS™ Complete 2-Channel Monitor            Each         1
    2  186-0224-AMS  BIS™ LOC 4-Channel Monitor with Patient Interface Cable (PIC-4)            Each         1
    3  186-0195-AMS          BIS™ LOC 2-Channel Monitor with Patient Interface Cable            Each         1
    4      186-0212                                            BIS™ Bilateral Sensor            Each         1
    
                  DISPLAY                                               BIS™ COMPLETE 2-CHANNEL MONITOR                                     BIS™ COMPLETE 4-CHANNEL MONITORING SYSTEM
    0          Parameters                                                    BIS, SQI, EMG, SR, BC, EEG                                           BIS, SQI, EMG, SR, BC, TP, SEF, EEG
    1  Trended parameters                                                         BIS, SQI, EMG, SR, BC                                                    BIS, SQI, EMG, SR, BC, SEF
    2          BIS™ alarm  Upper and lower limit, results in visual and audible alert when out of range  Upper and lower limit, results in visual and audible alert when out of range