I need a queuing system to help an application with processing different types of jobs. Example of jobs are:
There may be more in the future but that's the main part of it at the minute. I have searched high and low for examples of message queues but only found general information on how a message queue works (Kafka, RabbitMQ etc).
What I would like to know is because each type of job/message (video encoding, image resizing etc) is very distinct and has different meta information about each job (video will show codecs, resolution, bitrate whereas images have JPEG quality or transparency), are they be placed in different queues?
Or doesn't it matter and each job in a single queue would have unique metadata for that job?
I will answer regarding Kafka, although I think it's not that important in this context, so it won't really matter.
You can really write any messages in one Kafka topic, because the broker himself doesn't care, for him it's just a set of bytes. Serialization, deserialization, and parsing of the key, value, and headers will be handled by clients.
This suggests that it would be good to separate topics at least by some logical parameter, in your case different "jobs", and also provide certain schemes so that it is easier for clients to work when processing messages. Not to mention how we will expand the system with one universal "queue", sooner or later it will become a bottleneck. In general, message brokers are designed in such a way as to work with multiple queues (with partitions and logs in the case of Kafka).