I tried to write two functions in python which can be used for downloading data from sites and save those as csv files.
The functions are:
def get_site_url( site_number='01589440', date_from="2008-01-01",date_to="2017-12-31" ):
# input: site_number is a string
# the parameters in the url can have different formats. cb_00065 is the same as parameterCd=00060
#url = 'https://nwis.waterdata.usgs.gov/nwis/uv/cb_00065=on&format=rdb&site_no=%speriod=&begin_date=%s\
#&end_date=%s&siteStatus=all'%(site_number, date_from, date_to)
url = "https://nwis.waterdata.usgs.gov/nwis/uv/?parameterCd=00060,00065&format=rdb&site_no=%s&period=&begin_date=%s\
&end_date=%s&siteStatus=all"%(site_number, date_from, date_to)
return url
def download_data_from_url( url, savename='test.csv' ):
'''
One can download data with a different method that supports resume. If data is two large then it takes
lot of time and the connection to the server might be interrupted.
'''
buffer = BytesIO()
c = pycurl.Curl()
c.setopt(c.URL, url)
c.setopt(c.WRITEDATA, buffer)
c.perform()
c.close()
body = buffer.getvalue() # Body is a byte string.
with open( savename, 'w' ) as output:
output.write( body.decode('utf-8')) # We have to know the encoding in order to print it to a text file
Using this two function as next:
FirstExcUrl = get_site_url()
download_data_from_url(FirstExcUrl, '%s.csv'%'01589440')
produced the error:
errorTraceback (most recent call last)
<ipython-input-5-f460d226bec2> in <module>()
1 FirstExcUrl = get_site_url()
----> 2 download_data_from_url(FirstExcUrl, '%s.csv'%'01589440')
<ipython-input-3-e6e6e1be72e0> in download_data_from_url(url, savename)
19 c.setopt(c.URL, url)
20 c.setopt(c.WRITEDATA, buffer)
---> 21 c.perform()
22 c.close()
23
error: (1, 'Protocol "https" not supported or disabled in libcurl')
It is an output of a jupyter note book was started from an enthought Canopy editor window. I installed both curl and PycURL from Canpy package manager. The curl version got installed is: 7.58.0-1 and the version of PycURL got installed is: 7.19.5-5.
Any help, suggestion or solution is highly appreciated.
Sorry, we do not currently build libcurl with SSL support. You could use another python package (for example requests) which does support SSL.