rabbitmqactivemq-classicmessage-queueapplication-designweb-application-design

Message queue - architecture (task + response)


I am now thinking about using message queue for a "video" project.

Imaging a video service where users upload videos and then screenshots, short video thumbnail and different video sizes are made by scripts (it's not important how at the momemnt). Anyway, the more important thing to think about is the communication between frontend (web upload servers) and backend servers (video processing servers).

So proccess might be:

  1. User comes to the site
  2. User uploads new video
  3. Upload script add new message to message queue (hey, there's new video, process it)
  4. In this state user can see the video on a list of his videos with "Uploaded" status

This is the part I don't actually know how to do.

  1. Consumer gets message from the message queue and starts working on it.
  2. Consumer detects that it is video - if now it'll fail
  3. Consumer makes screenshots (takes a long time to do) and consumer will inform web that screenshots has been make (??? How ??? - Shall I put new message to the message queue?)
  4. Consumer makes video resizing (takes a long time to do) - the same question again - how shall consumer inform website, that video resizing is done?

So, the question is how to process messages from backend servers. I guess consumer will put another message to some log queue and other web consumer will handle them and update database. Is it right?


Solution

  • Just create two queues: one for the jobs, second for the responses.

    Process:

    Consumer of the second one can be everything - a process that writes the result to the database or - for example - a WebSocket server which sends a notification immediately.