multithreadinghazelcastexecutorservicehazelcast-imap

Hazelcast not using it's own partition-operations threads resulting in lack of order

I would like some support with the management of partition-aware operations. I am not able to see use of partition-operation threads within eclipse debugger (see attached images). Only ForkJoinPool workers are ever used.

This is applicable to the following operations:

  1. map.submitToKey(strategyId, new TriggerUpdateEntryProcessor(update)); This is using a Offloadable implementation (but not readonly) therefore should be processed in blocking mode, but it does not.
  2. executorService.executeOnKeyOwner(triggerUpdateTask, strategyId); It’s also meant to be possible to submit a partition-aware task https://docs.hazelcast.com/imdg/4.2/performance/data-affinity#partitionaware but this doesn’t work either.

Also, regardless of submitTo or executeOn, both act the same.

Should I see threads running under partition-operations? Is this a configuration issue.

I am assuming that executorService functionality is correct in that it’s only partition-aware in terms of running the thread on the right member but not running on a partition-operation thread, though would still expect it to run on a generic-operation thread.

This is confusing in the documentation as there are limited examples of partition operations and “partition” it seems is used to refer to both literal partition threading and member location.

• The only examples are IMap.get is and executor.submitToMember is not. But executor also has submitToKeyOwner so will get processed on the right partition but will it be by that member's generic-operation thread instead of partition-operation... • Data Affinity topic states "Data affinity ensures that related entries exist on the same member. If related data is on the same member, operations can be executed without the cost of extra network calls and extra wire data. This feature is provided by using the same partition keys for related data."

observation

idle threads

Solution

  • If you implement Offloadable the entry processor will NOT run on the partition thread. From the Javadoc:

    If the EntryProcessor implements the Offloadable interface the processing will be offloaded to the given ExecutorService allowing unblocking the partition-thread, which means that other partition-operations may proceed. The key will be locked for the time-span of the processing in order to not generate a write-conflict. In this case the threading looks as follows:

    partition-thread (fetch & lock)
    execution-thread (process)
    partition-thread (set & unlock, or just unlock if no changes)

    If you want to run the entry processor on the partition thread then just don't implement Offloadable.