apache-kafkaconfluent-platformaws-msk

AWS MSK vs Confluent for hosting Kafka?

In terms of investing for the most value, how does AWS MSK compare to Confluent when it comes to hosting an end to end Kafka event sourcing?

The main criteria to be used for comparing are:

  1. Deployment
  2. Infrastructure
  3. Monitoring
  4. Security
Solution

  • I have used open-source, on-prem Cloudera, and MSK. When comparing them together they have all had their quirks.

    Deployment / Infrastructure

    1. Based on the speed of provisioning a secure Kafka cluster, I think MSK would win hands down. Someone with Kafka, AWS Certificate Manager, and Terraform can get it all done very quickly. Though there are a few issues around Terraform TLS and AWS CLI but there are workarounds.

    2. If you are planning to use Kafka Connect then confluent makes lots of sense.

    3. If you have Kafka developers who have experience in writing Kafka Connect sinks and source, then you may not need a subscription-based model from Confluent. Though you may not save a lot of money, as you would either spend in development or spend in subscription costs.

    4. If you like serverless MSK is quite good. However, there is no SSH access to the Kafka cluster and you cannot tune the JVM.

    5. MSK also provides auto-scaling, but again if you are planning to use Terraform there may be some interoperability issues.

    Monitoring

    Monitoring is built out of the box for MSK via open monitoring via JMX metrics and prometheus. You also have CloudWatch as well, but open monitoring pretty much gives all the metrics you need. In open-source, you can easily deploy monitoring, rather MSK is doing the same.

    Security

    1. MSK provides security using either TLS or IAM. Though there are some issues around enabling IAM-based security for MSK using Terraform.
    2. Two-way TLS Client authentication is quite easy to set up.