rebusrebus-rabbitmqrebus-kafka

How much does "SetMaxParallelism" and "SetNumberOfWorkers" in Rebus mix up the order of processing messages from two topics?


I consistently send a lot of numbered messages to two topics:

for (var i=0; i<500; i++)
{
    await bus.Publish(new TestEvent1 { Order = i });
}
for (var i=0; i<500; i++)
{
    await bus.Publish(new TestEvent2 { Order = i });
}
  1. They are received by one worker in four streams and processed in different time, from fractions of seconds to a minute.
services.AddRebus((configurer, serviceProvider) => configurer
    ...
    .Options(o =>
    {
        o.SetMaxParallelism(4);
        o.SetNumberOfWorkers(1);
    });

Please tell me, how much will Rebus mix them during processing? The order of their processing can be mixed both between topics and inside topics?

  1. And if I accept the same events in the next configuration:
services.AddRebus((configurer, serviceProvider) => configurer
    ...
    .Options(o =>
    {
        o.SetMaxParallelism(1);
        o.SetNumberOfWorkers(4);
    });

Will the mixing be the same or different?

  1. And if I accept the same events in the next configuration:
services.AddRebus((configurer, serviceProvider) => configurer
    ...
    .Options(o =>
    {
        o.SetMaxParallelism(2);
        o.SetNumberOfWorkers(2);
    });

Will the mixing be the same or different?

  1. And if I accept the same events in the next configuration:
services.AddRebus((configurer, serviceProvider) => configurer
    ...
    .Options(o =>
    {
        o.SetMaxParallelism(1);
        o.SetNumberOfWorkers(2);
    });

Will the mixing be the same or different? Perhaps workers will consume events from their topics?

  1. And if I accept the same events in the next configuration:
services.AddRebus((configurer, serviceProvider) => configurer
    ...
    .Options(o =>
    {
        o.SetMaxParallelism(2);
        o.SetNumberOfWorkers(1);
    });

Will the mixing be the same or different? Perhaps threads will consume events from their topics?

If there is a test for certain rules for mixing the order of event processing, then please provide a link to this test.


Solution

  • Rebus will, in all cases, simply process messages in the order they're delivered from the input queue.

    If you have parallelism == 1 then messages will be processed in the order they're in in the queue(*).

    As soon as the parallelism > 1, there's a potential for handling more than one message concurrently, which makes it impossible to make any promises regarding order. If some of the messages take a long time to process, they will finish much later than their neighbors in the queue.

    So to answer all of your questions at once:

    One Rebus instance has one input queue. It will dequeue messages from the input queue and handle them as quickly as it can.

    If you have parallelism == 1, then it follows that messages will be handled in order(*).

    If you have parallelism > 1, then it follows that you cannot expect any ordering.


    (*): Please note that a FAILED MESSAGE can wreak havoc, if you assume messages arrive strictly in order. When a message fails (i.e. causes an exception to be thrown), Rebus will move the message to its dead-letter queue (named "error" by default). If you move the failed message back into Rebus' input queue, then it will probably be processed totally out-of-order.