architecturedistributed-system

Is a distributed queue suitable for building a scalable case assignment system?

This is the basic functionality I’m targeting.

  1. A new user registers with their details and uploads some documents. This creates a new case.
  2. Auditors login every morning and pick a pending case from the backlog.
  3. If all documents look good, auditors approve, otherwise reject the case.

My take

  1. /register POST method endpoint that runs a which captures the details from the user in a sharded MySQL database, and any files that they submitted in a BLOB storage (references updated in MySQL).
  2. Successful registration emits an event with the user ID to a distributed queue, for the time being I’m considering Amazon SQS but I want to find out if there are any open source alternatives that can prevent me from a vendor lock in.
  3. /audit POST method endpoint which the auditor will call to fetch one entry from the queue (SQS) and also update the status of this case, so other auditors don’t end up getting the same case. If they were able to obtain a lock for this case, good otherwise try again with the next item from the queue.
  4. /register and /audit would be separate services, so they can be scaled up and down independently.

I’m having second thoughts about introducing a queue in the first place. Am I better off just querying the database? Something like SELECT Id FROM USER WHERE status = ‘Pending’ FOR UPDATE

How would you build a system that can handle scale according to /registration usage and /auditor usage? How would you build a system that can handle dynamically scale according to registration load and auditor load?

Solution

  • I’m having second thoughts about introducing a queue in the first place. Am I better off just querying the database? Something like SELECT Id FROM USER WHERE status = ‘Pending’ FOR UPDATE

    Queues are usually introduced in automatic pipelines to desynchronize and decouple the single processing steps, but this does not seem to be your case. Also, at this level, a queue per se is very similar to a plain table, it mainly adds a notification system on top of it (locking you already get in SQL), so, if you can send an email or similar from your program, that on top of some dedicated tables should be all you need.

    So, I think you do are fine with a table (or rather, group of related tables) representing the registration data and approval state (the approval state definition itself coming from a state-transition diagram or similar that specifies the approval life-cycle).

    On the other hand, I would advice against just doing it all within the main USER table: certainly so if you want scalability, but I think this would be advisable in any case for clarity and manageability of such an implementation.

    How would you build a system that can handle dynamically scale according to registration load and auditor load?

    Saying "dynamically scale" is a bit improper, either the system is designed to scale, or it isn't: in other words, we may "dynamically scale" on e.g. server resources, not on application logic and code. That said, I would go with dedicated tables as explained above, which does scale.