apache-kafkaconfluent-platformconfluent-control-center

Confluent Control Center disk space requirements - why so large?


The system requirements for Confluent Control Center state that 500GB disk space is required. CCC appears to store some data in Kafka itself, so can someone please tell me what this storage is actually required for? i.e. what is preventing CCC from being deployed as a container in Kubernetes without persistent volume storage?


Solution

  • Yes, Control Center maintains some data back in Kafka for rolling aggregates, but it's using Kafka Streams to do that, which is configured with a RocksDB database, and can store that information on disk.

    The latest documentation says less than 500GB, though.

    Control Center relies on local state in RocksDB. We recommend at least 300 GB of storage space

    If you have used Control Center, you would know that you can access months of Kafka historical metrics.

    300 GB isn't really that much, when look at your average Oracle or SQL Server database at most companies.

    what is preventing CCC from being deployed as a container in Kubernetes without persistent volume storage?

    Other than being able to retain historical data, and have a slow start-up period on the pod rebooting and getting data, probably not much.

    8 cores and 32 GB of RAM to a container, though, might make one heavy pod.