C++: Multithreading and refcounted object

I'm currently trying to pass a mono threaded program to multithread. This software do heavy usage of "refCounted" objects, which lead to some issues in multithread. I'm looking for some design pattern or something that might solve my problem.

The main problem is object deletion between thread, normally deletion only decrement the reference counting, and when refcount is equal to zero, then the object is deleted. This work well in monothread program, and allow some great performance improvement with copy of big object.

However, in multithread, two threads might want to delete the same object concurrently, as the object is protected by a mutex, only one thread delete the object and block the other one. But when it releases the mutex, then the other thread continue its execution with invalid (freed object), which lead to memory corruption.

Here is an example with this class RefCountedObject

class RefCountedObject
{
public:
RefCountedObject()
:   _refCount( new U32(1) )
{}

RefCountedObject( const RefCountedObject& obj )
:   _refCount( obj._refCount )
{
    ACE_Guard< ACE_Mutex > guard( _refCountMutex );
    ++(*_refCount);
}

~RefCountedObject()
{
    Destroy();
}

RefCountedObject& operator=( const RefCountedObject& obj )
{
    if( this != &obj )
    {
        Destroy();
        ACE_Guard< ACE_Mutex > guard( _refCountMutex );
        _refCount = obj._refCount;
        ++(*_refCount);
    }

    return *this;
}

private:
    void Destroy()
    {
        ACE_Guard< ACE_Mutex > guard( _refCountMutex );  // thread2 are waiting here
        --(*_refCount);         // This cause a free memory write by the thread2
        if( 0 == *_refCount )
            delete _refCount;
    }

private:
    mutable U32* _refCount;
    mutable ACE_Mutex _refCountMutex; // BAD: this mutex only protect the refCount pointer, not the refCount itself
};

Suppose that two threads want to delete the same RefCountedObject, both are in ~RefCountedObject and call Destroy(), the first thread has locked the mutex and the other one is waiting. After the deletion of the object by the first thread, the second will continue its execution and cause a free memory write.

Anyone has experience with a similar problem and found a solution ?

---

Thanks all for your help, I realize my mistake: The mutex is only protecting refCount pointer, not the refCount itself! I've created a RefCount class which is mutex protected. The mutex is now shared between all refCounted object.

Now all works fine.

If the count is part of the object then you have an inherent problem if one thread can be trying to increase the reference count whilst another is trying to remove the last reference. There needs to be an extra value on the ref count for each globally accessible pointer to the object, so you can always safely increase the ref count if you've got a pointer.

One option would be to use boost::shared_ptr (see the docs). You can use the free functions atomic_load, atomic_store, atomic_exchange and atomic_compare_exchange (which are conspicuously absent from the docs) to ensure suitable protection when accessing global pointers to shared objects. Once your thread has got a shared_ptr referring to a particular object you can use the normal non-atomic functions to access it.

Another option is to use Joe Seigh's atomic ref-counted pointer from his atomic_ptr_plus project