In my understanding, when I want to send a movie (4GB) to a Kafka broker, one producer will send that 4GB byte of a video file (serialized it) and send it to a kafka broker and many consumers who want to see that movie will consume that movie file.
I heard Netflix uses Kafka to send and watch movies. I am curious how they use producer, broker, and consumer. I'm using Netflix, and it's really fast. I want to know how they use Kafka.(especially how they use producers and consumers)
And as far as I know, when sending a video file, you need to encode it, and serialize it to send the data. (maybe encoding is serializing in this case?) Did I understand correctly? If I am missing something, could you give me some tips and guidance?
Netflix uses Kafka as part of its centralized data lineage solution. It is not using Kafka to encode, stream video contents. You can read more about how Kafka is being used here.
Now to answer your question on why its video streaming services are so fast. You'll need to understand how Netflix leverages aws resources like ec2, s3 and others to create a highly scalable, fault-tolerant microservice architecture.
On top of this Netflix works with ISPs to localize contents using a program called Netflix Open Connect. This allows them to cache
the content locally which minimizes latency and saves on compute.