Ehcache Replicated Cache not synchronizing at startup

I have an ehcache Cache replicated across two machines. The peers correctly find each other and replicate once both peers are started. However, if the 1st peer starts first, and receives several elements, and then the 2nd peer starts later... the 2nd peer never sees the elements that were added while it was not yet alive.

Here is exactly the order:

1. Cache A is started
2. Add "1234" to Cache A
3. Cache B is started
4. get "1234" from Cache B -> NOT FOUND

My expectation: If 2 caches are replicated, then getting an existing element returns the same value for either cache.

My cache Elements are just primitive String/Integer types.

An example is in GitHub here: https://github.com/HamletDRC/EhcachePOC

Ehcache configurations are here: https://github.com/HamletDRC/EhcachePOC/tree/master/src/main/resources

In the sample project, log4j is enabled for the ehcache classes so that you can see that the peers do find each other and do replicate, but only elements that were added since the peer group started, not elements that existed previously.

You only need a JDK and Maven installed to build it.

To reproduce:

• Run ReplicatedCacheWriter
• Wait 6 seconds for the writer to create elements [1, 2, 3, 4, 5, 6]
• Run ReplicatedCacheListener
• The listener finds all elements that were "put" after it came alive, but no elements "put" before it came alive.

Here is the ehcache.xml

<cacheManagerPeerProviderFactory
        class="net.sf.ehcache.distribution.RMICacheManagerPeerProviderFactory"
        properties="peerDiscovery=automatic, multicastGroupAddress=231.0.0.1,
                  multicastGroupPort=4446, timeToLive=32"/>

<cacheManagerPeerListenerFactory
        class="net.sf.ehcache.distribution.RMICacheManagerPeerListenerFactory"
        properties="port=40002, socketTimeoutMillis=2000"/>

...

<cacheEventListenerFactory
            class="net.sf.ehcache.distribution.RMICacheReplicatorFactory"
            properties="replicateAsynchronously=true, replicatePuts=true, replicateUpdates=true,
                    replicateUpdatesViaCopy=false, replicateRemovals=true "/>

(Obviously, the Listener Port is different between the two nodes)

How can I get the caches to synchronize at startup?

You need a bootstrapCacheLoaderFactory in your ehcache_listener.xml. For example:

<cache name="myCache" ...>
   ...
   <bootstrapCacheLoaderFactory
            class="net.sf.ehcache.distribution.RMIBootstrapCacheLoaderFactory"
            properties="bootstrapAsynchronously=true"
            propertySeparator="," />    
</cache>