.netdockerapache-kafkaconfluent-schema-registrytestcontainers

Working with Kafka and Schema Registry containers together in dotnet TestContainers


I can't use the Kafka container along with the schema registry container in TestContainers. Here are my settings:

    private readonly KafkaContainer _kafkaContainer = new KafkaBuilder()
        .WithImage("confluentinc/cp-kafka:7.4.1")
        .WithHostname("broker")
        .WithName("broker")
        .WithNetworkAliases("broker")
        .WithNetwork(NetworkName)
        .WithPortBinding(9092, 9092)
        .WithWaitStrategy(Wait.ForUnixContainer().UntilPortIsAvailable(9092))
        .WithEnvironment(
            new ReadOnlyDictionary<string, string>(new Dictionary<string, string>
            {
                { "KAFKA_ADVERTISED_LISTENERS", "PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092" }
            }))
        .WithCleanUp(true)
        .Build();
    
    private readonly IContainer _schemaRegistryContainer = new ContainerBuilder()
        .WithImage("confluentinc/cp-schema-registry:7.4.1")
        .WithHostname("schema-registry")
        .WithName("schema-registry")
        .WithCleanUp(true)
        .WithEnvironment(
            new ReadOnlyDictionary<string, string>(new Dictionary<string, string>
            {
                { "SCHEMA_REGISTRY_HOST_NAME", "schema-registry" },
                { "SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS", "broker:29092" },
                { "SCHEMA_REGISTRY_LISTENERS", "http://0.0.0.0:8081" }
            }))
        .WithPortBinding(8081, 8081)
        .WithWaitStrategy(Wait.ForUnixContainer().UntilPortIsAvailable(8081))
        .WithNetwork(NetworkName)
        .Build();

What I see on logs of schema registry container:

2023-12-21 13:27:21 [2023-12-21 12:27:21,315] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (broker/172.27.0.2:29092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
2023-12-21 13:27:22 [2023-12-21 12:27:22,364] INFO [AdminClient clientId=adminclient-1] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient)
2023-12-21 13:27:22 [2023-12-21 12:27:22,364] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (broker/172.27.0.2:29092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
2023-12-21 13:27:23 [2023-12-21 12:27:23,604] INFO [AdminClient clientId=adminclient-1] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient)
2023-12-21 13:27:23 [2023-12-21 12:27:23,604] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (broker/172.27.0.2:29092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
2023-12-21 13:27:24 [2023-12-21 12:27:24,741] INFO [AdminClient clientId=adminclient-1] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient)
2023-12-21 13:27:24 [2023-12-21 12:27:24,742] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (broker/172.27.0.2:29092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
2023-12-21 13:27:25 [2023-12-21 12:27:25,772] INFO [AdminClient clientId=adminclient-1] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient)

What should I change to make them work together successfully?

Thanks,


Solution

  • So I'm not 100% understand why it has to be this, and why the other config doesn't work. But this is the setup that starts both containers successfully:

        private readonly static INetwork network = new NetworkBuilder()
            .WithName(Guid.NewGuid().ToString("D"))
            .Build();
      
        private static readonly KafkaContainer _kafkaContainer = new KafkaBuilder()
            .WithImage("confluentinc/cp-kafka:7.4.1")
            .WithNetworkAliases("kafka")
            .WithNetwork(network)
            .WithEnvironment(new ReadOnlyDictionary<string, string>(new Dictionary<string, string>
            {
                {"KAFKA_CONFLUENT_SCHEMA_REGISTRY_URL", "http://schemaregistry:8085"}
            }))
            .WithCleanUp(true)
            .Build();
        
        private static readonly IContainer _schemaRegistryContainer = new ContainerBuilder()
            .WithImage("confluentinc/cp-schema-registry:7.4.1")
            .WithCleanUp(true)
            .WithEnvironment(
                new ReadOnlyDictionary<string, string>(new Dictionary<string, string>
                {
                    { "SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS", "PLAINTEXT://kafka:9093" },
                    { "SCHEMA_REGISTRY_KAFKASTORE_SECURITY_PROTOCOL", "PLAINTEXT"},
                    { "SCHEMA_REGISTRY_LISTENERS", "http://0.0.0.0:8085" },
                    { "SCHEMA_REGISTRY_HOST_NAME", "schemaregistry" }
                }))
            .WithPortBinding(8085, true)
            .WithWaitStrategy(Wait.ForUnixContainer().UntilMessageIsLogged("Server started, listening for requests"))
            .WithNetwork(network)
            .WithNetworkAliases("schemaregistry")
            .DependsOn(_kafkaContainer)
            .Build();
    

    Note that the KafkaBuilder already exposes the ports, and does the configuration for waiting and other details like that. So we only add the network info and tell it to expect a schema registry.

    The schema registry container, for which we don't have a module (but maybe actually should, or like have an extension of the Kafka module to add the Schema Registry container to the setup), needs more config.

    I also changed the ports to mirror my example in Java that was shared in the comments.