UPDATE statement on table xxx' expected to update 1 row(s); 0 were matched with Zope transactionmanager

I am running a Pyramid + Zope transaction manager + SQLAlchemy + PostgreSQL. On some occasions, I have seen StaleDataError error on a Pyramid web application which should very trivial view of updating one row in a database. As the error happens outside the normal view boundary and is not repeatable, it is quite tricky to debug.

I guess this might have something to do with broken database connections or transaction lifecycle. However I don't know how to start debugging the system, so I am asking what could cause this and furthermore how one can pin down errors like this.

UPDATE statement on table 'xxx' expected to update 1 row(s); 0 were matched.

Stacktrace (most recent call last):

  File "pyramid/tweens.py", line 20, in excview_tween
    response = handler(request)
  File "pyramid_tm/__init__.py", line 94, in tm_tween
    reraise(*exc_info)
  File "pyramid_tm/compat.py", line 15, in reraise
    raise value
  File "pyramid_tm/__init__.py", line 82, in tm_tween
    manager.commit()
  File "transaction/_manager.py", line 111, in commit
    return self.get().commit()
  File "transaction/_transaction.py", line 280, in commit
    reraise(t, v, tb)
  File "transaction/_compat.py", line 55, in reraise
    raise value
  File "transaction/_transaction.py", line 271, in commit
    self._commitResources()
  File "transaction/_transaction.py", line 417, in _commitResources
    reraise(t, v, tb)
  File "transaction/_compat.py", line 55, in reraise
    raise value
  File "transaction/_transaction.py", line 389, in _commitResources
    rm.tpc_begin(self)
  File "/srv/pyramid/trees/venv/lib/python3.4/site-packages/zope/sqlalchemy/datamanager.py", line 90, in tpc_begin
    self.session.flush()
  File "sqlalchemy/orm/session.py", line 2004, in flush
    self._flush(objects)
  File "sqlalchemy/orm/session.py", line 2122, in _flush
    transaction.rollback(_capture_exception=True)
  File "sqlalchemy/util/langhelpers.py", line 60, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "sqlalchemy/util/compat.py", line 182, in reraise
    raise value
  File "sqlalchemy/orm/session.py", line 2086, in _flush
    flush_context.execute()
  File "sqlalchemy/orm/unitofwork.py", line 373, in execute
    rec.execute(self)
  File "sqlalchemy/orm/unitofwork.py", line 532, in execute
    uow
  File "sqlalchemy/orm/persistence.py", line 170, in save_obj
    mapper, table, update)
  File "sqlalchemy/orm/persistence.py", line 692, in _emit_update_statements
    (table.description, len(records), rows))

This is most likely scenario:

You have 2 requests that first select an object and try to update/delete it in the datastore and you end up with a "race condition".

Lets say you want to do something like fetch an object and then update it.

If transaction takes some time and you do not select the object with "for update" thus locking the rows - if the object gets deleted in first request and 2nd transaction tries to issue update to row that is not present in the db anymore you can end up with this exception.

You can try doing some row locking to prevent this from happening - subsequent transaction will "wait" for the first operation to finish. Before it gets executed.

http://docs.sqlalchemy.org/en/rel_1_0/orm/query.html?highlight=for_update#sqlalchemy.orm.query.Query.with_for_update

and

http://docs.sqlalchemy.org/en/rel_1_0/orm/query.html?highlight=with_lockmode#sqlalchemy.orm.query.Query.with_lockmode

Describe some of the sqlalchemy machinery you can use to resolve this.