python-3.xcookiesurllibcookiejar

How to save urllib python3 cookies to a file


Trying to store/re-use cookies between code-executions, similar to this question, but using only python3's urllib.

Thanks to this answer for the process for creating a cookiejar for automatic use across urllib.request.Request calls:

cookie_jar = http.cookiejar.CookieJar()
opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cookie_jar))
urllib.request.install_opener(opener)

Initial searches on cookie storage led to SO questions regarding requests (the module) + cookies, but unfortunately, http.cookiejar objects cannot be pickled (below), as opposed to their requests brethren, which can:

>>> pickle.dumps(cookie_jar)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: can't pickle _thread.RLock objects

The other queries' results mostly pointed back the python3 docs: https://docs.python.org/3/library/http.cookiejar.html#http.cookiejar.FileCookieJar

Playing around with the FileCookieJar class led to noticing in the description: A CookieJar which can load cookies from, and *perhaps* save cookies to, a file on disk. and the associated error:

>>> FCJ=cookiejar.FileCookieJar("unversioned.cookies")              
>>> FCJ.save()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.7/http/cookiejar.py", line 1785, in save
    raise NotImplementedError()

Both of which were clarified by this answer - which suggested using LWPCookieJar (among others) instead, but I couldn't find any existing examples on how to use this class in practice.


Solution

  • LWPCookieJar was the answer:

    1. Create the CookieJar + try loading from any existing cookie file:
    cookie_filename = "unversioned.cookies"
    cookie_jar = cookiejar.LWPCookieJar(cookie_filename)
    try:
        cookie_jar.load()
    except FileNotFoundError as fnfe:
        # No existing/adjacent cookie file
        pass
    
    opener = request.build_opener(request.HTTPCookieProcessor(cookie_jar))
    request.install_opener(opener)
    
    1. complete any urllib.request.Request call that populates cookies
    2. store the cookies to file (named in the constructor):
    cookie_jar.save()
    

    And to confirm, after having saved some cookies in a previous execution-run:

    1. prepare the cookiejar, but do not call .load()
    2. test-request the page -> logged-out version received
    3. call cookiejar.load()
    4. test-request the page -> logged-in version received