[SOLVED] Memcached-based beaker sessions not initializing on Zope start

Memcached-based beaker sessions not initializing on Zope start

We're using memcached-based beaker sessions for a distributed Plone/Zope setup. The same setup works fine in our pre-prod environments but when we move to prod, we can't seem to get it to connect to memcached on 11211.

I've run tcpdump on the machine while starting Zope and it isn't even attempting a connection.

I've telnet from the Zope server to the memcached server (same as ZEO server) to test the connection and that works fine.

I've tried two different prod Zope servers and the result is the same.

We have our old setup (4.2.5) running on yet another prod server and when we start it, we get the expected behavior and it connects to memcached just fine. Unfortunately, we also have a 4.3.2 setup (same as prod) running in pre-prod and it ALSO works just fine. I can't identify any substantial differences in the setups.

All systems are running isolated (not system/apt controlled) Python 2.7s.

Relevant piece of zope.conf

# Beaker Configuration
zope-conf-additional =
<product-config beaker>
    cache.type              ext:memcached
    cache.url               ${ips:memcached}
    cache.data_dir          ${buildout:directory}/var/cache/data
    cache.lock_dir          ${buildout:directory}/var/cache/lock
    cache.regions           short, long
    cache.short.expire      60
    cache.long.expire       3600

    session.type            ext:memcached
    session.url             ${ips:memcached}
    session.data_dir        ${buildout:directory}/var/sessions/data
    session.lock_dir        ${buildout:directory}/var/sessions/lock
    session.key             beaker.session
    session.secret          magicalsecretcornedbeefhash
</product-config>

zcml = collective.beaker

Relevant versions from versions.cfg:

# Beaker
Beaker = 1.6.4
Products.BeakerSessionDataManager = 1.1
collective.beaker = 1.0b3
python-memcached = 1.47

I've tested that ${ips:memcached} is producing the correct value on startup. No errors are logged to the Zope log, but no beaker sessions are created and no connection attempts made to memcached. Memcached is happily running but produces no logs (seems to never have).

Any ideas or advice on what could possibly be causing this is helpful. More background:

Ubuntu 12.04.4 LTS
Linux 3.2.0-58-generic #88-Ubuntu SMP Tue Dec 3 17:37:58 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

bin/buildout:

#!/opt/zope/pythons/python-2.7/bin/python

import sys
sys.path[0:0] = [
    '/opt/zope/buildouts/FOO/eggs/distribute-0.6.28-py2.7.egg',
    '/opt/zope/buildouts/FOO/eggs/zc.buildout-1.7.1-py2.7.egg',
]

Solution

When moving to production, remember the last part in the Products.BeakerSessionDataManager README:

3. In the ZMI, delete the ``session_data_manager`` object and add a
   ``Beaker Session Data Manager``.