kubernetesevent-handlingmicroservicesreplicasetnats-streaming-server

How to handle multiple update event when there is more then one replica of a pod


I have two services name Product and Order. Order table inside OrderDb has price and productId columns for storing product price and product id that ordered. Order services has 3 replica.

Now, suppose a product is ordered and it's id 80, and a series of sequential update event fired from product service to order services for that particular product:

event1:{productId: 80, price: 500}
event2:{productId: 80, price: 600}
event3:{productId: 80, price: 400}
event4:{productId: 80, price: 900}
event5:{productId: 80, price: 100}

so the end price should be 100 for that product, but sometimes these events are processed in random order such as

event1:{productId: 80, price: 500}
event2:{productId: 80, price: 600}
event5:{productId: 80, price: 100}
event4:{productId: 80, price: 900}
event3:{productId: 80, price: 400}

since event 3 processed last, price become 400.


Solution

  • This is generally up to your database. I see you put NATS in the tags so I'm assuming you mean you have some kind of worker queue model, but you probably have a database of record behind that with its own consistency model. With event streaming systems where you want to defend against out of order or multi-delivery, you would include more info in the queue message such as an object version, or just the previous price. In the latter, simpler case it would be like

    event1:{productId: 80, price: 500, oldPrice: 0}
    event2:{productId: 80, price: 600, oldPrice: 500}
    event3:{productId: 80, price: 400, oldPrice: 600}
    event4:{productId: 80, price: 900, oldPrice: 400}
    event5:{productId: 80, price: 100, oldPrice: 900}
    

    That would let your code refuse to apply the operation if the base state no longer matches. But that's pretty limiting, you don't want everything to fail after a re-order, you just want convergent behavior. This the point where I just yell "VECTOR CLOCKS" and leap out of the window. Designing distributed, convergent systems is indeed quite hard, look up the term CRDT as a starting point.