@vary_on_cookie fails due to non-Django cookies

I am stumped on a caching issue in my Django 1.5.6 application:

@vary_on_cookie
@cache_page(24 * 60 * 60, key_prefix=':1:community')
@rendered_with("general/community.html")
@allow_http("GET")
def community(request):
    ...
    return { ... }

Locally the caching is working correctly, but when I test this in staging, @vary_on_cookie isn't working -- I can see by the queries being executed that community() is being executed on subsequent calls to this page.

I updated my settings in my local environment to use the same Redis cache as staging to eliminate that difference, but the local environment continued to behave correctly.

Looking at the keys Redis has in its cache, I can see what the problem is -- in staging every time this page gets called, new keys are added to the cache. Compare the output from cache.keys('*community*'):

LOCAL:

First call to community page:

[u'community:1:views.decorators.cache.cache_page.:1:community.GET.b528759dd79cf1c6b405290c0bc05e39.3b7d4c38ec8d92512a4a0847f4738298.en-us.America/New_York', 
u'community:1:views.decorators.cache.cache_header.:1:community.b528759dd79cf1c6b405290c0bc05e39.en-us.America/New_York']

Second call (same user):

[u'community:1:views.decorators.cache.cache_page.:1:community.GET.b528759dd79cf1c6b405290c0bc05e39.3b7d4c38ec8d92512a4a0847f4738298.en-us.America/New_York', 
u'community:1:views.decorators.cache.cache_header.:1:community.b528759dd79cf1c6b405290c0bc05e39.en-us.America/New_York']

Notice there are the same number of keys in both cases.

STAGING:

First call to community page:

[u'community:1:views.decorators.cache.cache_header.:1:community.b528759dd79cf1c6b405290c0bc05e39.en-us.America/New_York',     
u'community:1:views.decorators.cache.cache_page.:1:community.GET.b528759dd79cf1c6b405290c0bc05e39.559380b85dc0cdcf0ff25051df78987d.en-us.America/New_York']

Second call (same user):

[u'community:1:views.decorators.cache.cache_header.:1:community.b528759dd79cf1c6b405290c0bc05e39.en-us.America/New_York', 
u'community:1:views.decorators.cache.cache_page.:1:community.GET.b528759dd79cf1c6b405290c0bc05e39.559380b85dc0cdcf0ff25051df78987d.en-us.America/New_York', 
u'community:1:views.decorators.cache.cache_page.:1:community.GET.b528759dd79cf1c6b405290c0bc05e39.6ec85abcc8a14d66800228bdccc537f0.en-us.America/New_York']

Notice that an additional entry has been added to the cache though it's the same user!

I'm stumped where to go from here. Both environments are using SESSION_ENGINE = 'django.contrib.sessions.backends.cached_db'. The staging environment clearly recognizes that this is the same user in every other way. What is happening in @vary_on_cookie that is creating a difference in staging, but not locally?

I've inspected all of my staging vs. local differences, scrutinized my custom middleware, but I don't have any ideas of what to look at. Any ideas even of what to look at next would be greatly appreciated. Thanks!

UPDATE

I inspected django.utils.cache._generate_cache_key() to see how it generates that last hex section of the cache key. I naively assumed it just looked at Django's own cookies (like sessionid), but I see that it uses all of the cookies passed into HTTP_COOKIE -- that means, Django and non-Django. For me, that means cookies from Google Analytics and New Relic, neither of which I have running locally.

for header in headerlist:  # headerlist = [u'HTTP_COOKIE']
        value = request.META.get(header, None) # the string of all cookies, for ex: __atuvc=39%7C17%2C8%7C18; csrftoken=dPqaXS6XVGp2UUvfhEW9kS6R6WPHQlE4; sessionid=j6a83wbsq1sez9bz75n0tzl4n884umg2'
        if value is not None:
            ctx.update(force_bytes(value))

Can this really be true?! All of the world's Django sites using @vary_on_cookie are being thwarted by their third-party cookies?!

Solution

I created a custom decorator which hacks the HTTP headers to isolate the user's ID. (Although it sets Vary: DJANGO_USERID, Cookie in the response sent back to the browser, it doesn't include the actual ID.)

I would appreciate any feedback on this solution, since it's a bit beyond my Django comfort zone.

def vary_on_user(view):
    """                                                                                                                                                                                                     
    Adapted from django.views.decorators.vary_on_cookie                                                                                                                                                                                          
    """
    @wraps(view, assigned=available_attrs(view))
    def inner_func(request, *args, **kwargs):
        request.META['HTTP_DJANGO_USERID'] = request.user.id
        response = view(request, *args, **kwargs)
        patch_vary_headers(response, ('DJANGO_USERID',))
        return response
    return inner_func