Pyramid app "Header names must be latin1 string" AssertionError from WebTest

How can I deal with a "Header names must be latin1 string" AssertionError from WebTest that occurs when Pyramid apps are run in Python 2 with from __future__ import unicode_literals?

We're in the process of migrating our Pyramid app from Python 2 to 3, and added from __future__ import unicode_literals to all our Python files, and this caused the following error from WebTest in Python 2:

AssertionError: Header names must be latin1 string (not Py2 unicode or Py3 bytes type).

Here's a full traceback in case anyone's interested, though I don't think it's very enlightening.

The AssertionError is raised whenever your app sets any response header. For example this line would trigger it:

response.headers["Access-Control-Allow-Origin"] = "*"

Since we have unicode_literals those string literals are unicode strings in Python 2, whereas before unicode_literals they would have been byte strings, that's why adding unicode_literals triggered the AssertionError from WebTest.

In Python 3 no error occurs.

The reason WebTest has this AssertionError is that https://www.python.org/dev/peps/pep-3333 requires HTTP response headers to be native strings - byte strings in Python 2 and unicode strings in Python 3. Here's the WebTest issue and pull request that added the assert:

https://github.com/Pylons/webtest/issues/119
https://github.com/Pylons/webtest/pull/180

b-prefixing the strings like response.headers[b"Access-Control-Allow-Origin"] = b"*" would get rid of the AssertionError in Python 2 but cause the error to appear if the tests were run in Python 3.

Wrapping the strings in str() like response.headers[str("Access-Control-Allow-Origin")] = str("*") will fix it in both Python 2 and 3, but requires you to find and str()-wrap every response header string throughout your app.

Adding a tween that str()-wraps all response headers seems to be a good fix:

def encode_headers_tween_factory(handler, registry):
    def encode_headers_tween(request):
        resp = handler(request)
        for key in resp.headers.keys():
            values = resp.headers.getall(key)
            del resp.headers[key]
            for value in values:
                resp.headers.add(str(key), str(value))
        return resp
    return encode_headers_tween

config.add_tween('h.tweens.encode_headers_tween_factory')

Then you can simply remove this tween once you no longer need to support Python 2.