pythoncookiespython-requestsurlretrieve

Adding cookie data into a requests.urlretrieve


i am trying to download a .torrent file from a password protected site. I have managed to get to the site using cookies like so:

cookies = {'uid': '232323', 'pass': '31321231jh12j3hj213hj213hk',
           '__cfduid': 'kj123kj21kj31k23jkl21j321j3kl213kl21j3'}
    try:
        # read site content
        read = requests.get(s_string, cookies=cookies).content
    except RequestException as e:
        raise print('Could not connect to somesite: %s' % e)

    soup = BeautifulSoup(read, 'html.parser')

With this above code i get access to the site and scrape the data i need. With the scraped data i build a link to a .torrent file, which i then want to download, but this is where i am stuck.

Here is what im trying right now: (cookie data not real obviously, like its not in above code either)

cookies = {'uid': '232323', 'pass': '31321231jh12j3hj213hj213hk',
               '__cfduid': 'kj123kj21kj31k23jkl21j321j3kl213kl21j3'}

# construct download URL
torrent_url = ('https://www.somesite.com/' + torrent_url)
# for testing purposes DELETE!
print('torrent link:', torrent_url)

# download torrent file into a folder
filename = torrent_url.split('/')[-1]
save_as = 'torrents/' + filename + '.torrent'

try:
    r = request.urlretrieve(torrent_url, save_as, data=cookies)
    print("Download successful for: " + filename)
except request.URLError as e:
        raise print("Error :%s" % e)

This code would work without the cookies on a normal site, but this .torrent file im trying to get is behind a passworded/captchaed site, so i need to use cookies to scrape it.

So question is, what am i doing wrong here? without data=cookies i get http 404 error and with the data=cookies i get the following error:

File "/usr/lib/python3.6/http/client.py", line 1064, in _send_output
+ b'\r\n'
TypeError: can't concat str to bytes </error>

ps. before anyone asks, yes im 100% sure the torrent_url is correct, i have it printed and manually copy/pasting it into my own browser promps the download window for the .torrent file in question

EDIT:

try:
   read = requests.session().get(torrent_url)
   with open(save_as, 'wb') as w:
       for chunk in read.iter_content(chunk_size=1024):
           if chunk:
               w.write(chunk)
           w.close()
           print("Download successful for: " + filename)
 except request.URLError as e:
     print("Error :%s" % e)

made this based on furas's suggestion, it works now, but when i try to open the .torrent, torrent client says "invalid coding, cannot open".

When i open the .torrent file, inside is this:

<h1>Not Found</h1>
<p>Sorry pal :(</p>
<script src="/cdn-cgi/apps/head/o1wasdM-xsd3-9gm7FQY.js"></script>

am i still doing something wrong or has this something to do with the site owner preventing programs from downloading .torrents from his site or something of that nature?


Solution

  • This works, but not ideal i think.

    cookies = {'uid': '232323', 'pass': '31321231jh12j3hj213hj213hk',
               '__cfduid': 'kj123kj21kj31k23jkl21j321j3kl213kl21j3'}
    
    try:
        read = requests.get(torrent_url, cookies=cookies)
        with open(save_as, 'wb') as w:
            for chunk in read.iter_content(chunk_size=512):
                if chunk:
                    w.write(chunk)
                print(filename + ' downloaded successfully!!!')
    except request.URLError as e:
        print("Error :%s" % e)