pythonbeautifulsoupscreen-scrapinghtml-frames

get the data from html frames

I try to get the tables information in this html frames. I mean the tables having columns:

Year,Month,Oil Production m3,Gas Production Ksm3,...

Using beautifulSoup, This is so far what I have tried:

from bs4 import BeautifulSoup
from urllib import urlopen, urlretrieve, quote

url_base = 'https://www.og.decc.gov.uk/information/wells/pprs/Well_production_onshore_oil_fields/onshore_oil_fields_by_well/onshore_oil_fields_by_wel.html'
u = urlopen(url_base)
html = u.read().decode('utf-8')
u.close()
soup = BeautifulSoup(html)

But this retrieve only the main page info, not the page frames. When I change the url base by the frame link , it tells me that the requested page is outdated.

Solution

  • I think you are copying the wrong url. When I use the following, it worked.

    url_base = 'https://www.og.decc.gov.uk/information/wells/pprs/Well_production_onshore_oil_fields/onshore_oil_fields_by_well/0.htm'

    Attention: it is .../onshore_oil_fields_by_well/0.htm,

    instead of .../onshore_oil_fields_by_well/0.html