pythonpandascsvbuttonarcgis

How to scrape a csv table contained within a JavaScript button using Python?


I am trying to pull a csv table from a webpage into a pandas dataframe within a Jupyter Notebook, a process I have successfully implemented several times for other websites in the past, but I am running into a challenge with this one.

The table I am trying to pull is only available as a JavaScript response to a button click instead of being available at a specific URL.

Here is the URL: https://ciffc.net/national // The table I am trying to scrape is contained within the Export (CSV) button in red.

I have tried implementing a solution using pandas.read_csv, but I can't seem to point to the button in question.

Of note, I am using Jupyter Notebooks within the ArcGIS Online environment.

Thank you!


Solution

  • The json for example for page 1 seems to be available over the stable URL https://api.ciffc.net//v1/wildfires?page=1.

    So if you don't mind doing some extra formatting on the DataFrame you can use:

    import pandas
    import requests
    
    df = pandas.DataFrame.from_dict(
        requests.get('https://api.ciffc.net//v1/wildfires?page=1').json()["rows"])
    

    Which contains:

    field_agency_fire_id field_agency_code field_agency_data_timezone field_situation_report_date field_status_date field_stage_of_control_status field_system_fire_cause field_response_type field_fire_size field_latitude field_longitude full_count
    0 SS007-24 nt MDT 2024-05-12T22:21:00.000Z 2024-06-14T20:00:10.000Z OC N MON 1 60.5853 -116.902 1515
    1 FS005-24 nt MDT 2024-05-13T17:40:00.000Z 2024-06-14T20:00:10.000Z UC N FUL 22 61.52 -121.393 1515
    2 SS015-24 nt MDT 2024-06-01T21:57:00.000Z 2024-06-14T20:00:10.000Z OC N FUL 0.2 60.0809 -117.185 1515
    3 SS008-24 nt MDT 2024-05-23T23:24:00.000Z 2024-06-14T20:00:10.000Z OC N MON 8 60.2382 -118.13 1515
    4 SS003-24 nt MDT 2024-05-05T19:45:00.000Z 2024-06-14T20:00:10.000Z OUT N FUL 0.01 60.6161 -116.073 1515
    5 FS006-24 nt MDT 2024-05-13T21:47:00.000Z 2024-06-14T20:00:10.000Z OC N MON 2930 61.2465 -119.392 1515
    6 ZF004-24 nt MDT 2024-05-25T17:55:00.000Z 2024-06-14T20:00:10.000Z OUT H FUL 0 62.4502 -114.401 1515
    7 ZF001-24 nt MDT 2024-02-05T15:00:00.000Z 2024-06-14T20:00:10.000Z OUT H FUL 0.05 62.3217 -114.252 1515
    8 SS012-24 nt MDT 2024-05-25T18:00:00.000Z 2024-06-14T20:00:10.000Z OUT H FUL 0.1 61.1722 -113.669 1515
    9 SS009-24 nt MDT 2024-05-24T16:08:00.000Z 2024-06-14T20:00:10.000Z OC N MOD 5681 60 -117 1515
    10 SS005-24 nt MDT 2024-05-10T20:10:00.000Z 2024-06-14T20:00:10.000Z OUT H FUL 0.5 60.5885 -116.176 1515
    11 SS004-24 nt MDT 2024-05-05T20:45:00.000Z 2024-06-14T20:00:10.000Z BH N FUL 3 60.3924 -117.896 1515
    12 SS002-24 nt MDT 2024-04-23T06:00:00.000Z 2024-06-14T20:00:10.000Z OUT N FUL 0.5 60.3258 -112.282 1515
    13 SS001-24 nt MDT 2024-04-23T06:00:00.000Z 2024-06-14T20:00:10.000Z OUT N FUL 2 60.119 -112.281 1515
    14 ZF007-24 nt MDT 2024-06-06T22:37:00.000Z 2024-06-14T20:00:10.000Z OUT H FUL 0 62.5128 -113.694 1515
    15 ZF005-24 nt MDT 2024-05-26T13:00:00.000Z 2024-06-14T20:00:10.000Z OUT H FUL 0 62.5185 -114.319 1515
    16 ZF003-24 nt MDT 2024-05-23T23:30:00.000Z 2024-06-14T20:00:10.000Z OUT H FUL 5 63.4575 -116.098 1515
    17 ZF002-24 nt MDT 2024-05-19T23:45:00.000Z 2024-06-14T20:00:10.000Z OUT H FUL 0.01 62.4923 -113.521 1515
    18 SS014-24 nt MDT 2024-05-29T23:55:00.000Z 2024-06-14T20:00:10.000Z OUT N FUL 2.5 60.2638 -116.334 1515
    19 SS013-24 nt MDT 2024-05-27T20:18:00.000Z 2024-06-14T20:00:10.000Z OUT N FUL 0.5 60.3 -116.65 1515
    20 SS011-24 nt MDT 2024-05-24T18:40:00.000Z 2024-06-14T20:00:10.000Z OUT N FUL 1 60.9028 -112.421 1515
    21 SS010-24 nt MDT 2024-05-24T18:22:00.000Z 2024-06-14T20:00:10.000Z OC N MOD 28 60.9536 -112.39 1515
    22 ZF006-24 nt MDT 2024-05-31T21:49:00.000Z 2024-06-14T20:00:10.000Z OC N MON 145 62.5954 -119.058 1515
    23 SS006-24 nt MDT 2024-05-10T21:49:00.000Z 2024-06-14T20:00:10.000Z OUT N MON 1 60.6623 -115.978 1515
    24 FS004-24 nt MDT 2024-05-10T18:20:00.000Z 2024-06-14T20:00:10.000Z OC N MOD 11894 61.1473 -119.376 1515