Can't connect to ScyllaDB containers from host

I've just tried to connect to a 3-node Scylla cluster spun up on Docker Desktop from my local machine with Golang without success (on a Mac M2)

I tried to follow the steps here: https://university.scylladb.com/courses/using-scylla-drivers/lessons/golang-and-scylla-part-1/

The exception is that I didn't create a docker container for the test app cause I don't want to recreate a container any time I modify the code. I want to run the test locally for now.

docker-compose part

services:
  scylla-node1:
    image: scylladb/scylla:latest
    restart: always
    container_name: scylla-node1
    command: --seeds=scylla-node1,scylla-node2 --smp 1 --memory 750M --overprovisioned 1 --api-address 0.0.0.0
    volumes:
      - './scylla/scylla.yaml:/etc/scylla/scylla.yaml'
      - './scylla/cassandra-rackdc.properties.dc1:/etc/scylla/cassandra-rackdc.properties'
      - './scylla/mutant-data.txt:/mutant-data.txt'
    networks:
      - myappnetwork

  scylla-node2:
    image: scylladb/scylla:latest
    restart: always
    container_name: scylla-node2
    command: --seeds=scylla-node1,scylla-node2 --smp 1 --memory 750M --overprovisioned 1 --api-address 0.0.0.0
    volumes:
      - './scylla/scylla.yaml:/etc/scylla/scylla.yaml'
      - './scylla/cassandra-rackdc.properties.dc1:/etc/scylla/cassandra-rackdc.properties'
    networks:
      - myappnetwork

  scylla-node3:
    image: scylladb/scylla:latest
    restart: always
    container_name: scylla-node3
    command: --seeds=scylla-node1,scylla-node2 --smp 1 --memory 750M --overprovisioned 1 --api-address 0.0.0.0
    volumes:
      - './scylla/scylla.yaml:/etc/scylla/scylla.yaml'
      - './scylla/cassandra-rackdc.properties.dc1:/etc/scylla/cassandra-rackdc.properties'
    networks:
      - myappnetwork

networks:
  myappnetwork:
    driver: bridge

Then, I created the keyspace and data as explained in the tutorial

And tried to connect with gocql:

// go.mod

go 1.21.0

replace github.com/gocql/gocql => github.com/scylladb/gocql v1.11.1


import (
  ...
  "github.com/gocql/gocql"
)

func Start(log logger.Logger) {
    config := CreateCluster(gocql.Quorum, "catalog", "scylla-node1", "scylla-node2", "scylla-node3")
    // config := CreateCluster(gocql.Quorum, "catalog", "172.29.0.5", "172.29.0.4", "172.29.0.3")
    // >>> doesn't work either cause I can't even ping containers IPs
    session, err := gocql.NewSession(*config)
    if err != nil {
        log.Fatal(err, "db", "Start", "unable to connect to scylla")
    }
    defer session.Close()
}

func CreateCluster(consistency gocql.Consistency, keyspace string, hosts ...string) *gocql.ClusterConfig {
    retryPolicy := &gocql.ExponentialBackoffRetryPolicy{
        Min:        time.Second,
        Max:        10 * time.Second,
        NumRetries: 5,
    }
    config := gocql.NewCluster(hosts...)
    config.PoolConfig.HostSelectionPolicy = gocql.TokenAwareHostPolicy(gocql.RoundRobinHostPolicy())
    config.Compressor = &gocql.SnappyCompressor{}
    config.RetryPolicy = retryPolicy
    config.Timeout = 5 * time.Second

    config.Keyspace = keyspace
    config.Consistency = consistency

    return config
}

Error gocql: unable to create session: unable to discover protocol version: dial tcp :0->172.29.0.3:9042: i/o timeout

Solution

You are right, but that's how the underlying Docker implementation works on a Mac. :-)

Containers run inside a VM and - due to that -, and you can't route traffic to the container IP addresses by default.

You can, however, bind ports from a container to the host machine, but the problem of doing so is that your driver will be unable to connect given how the CQL protocol works.

You have a few options:

1 - Spin up a single ScyllaDB container, bind port 9042/19042 (and/or the relevant TLS ones, if applicable) to your host machine. Tell your application to connect to localhost and done. This approach won't work for a cluster of more than 1 node.

2 - Move your application to a container under the same network. As the container will live in the same network as the underlying VM, traffic will be routable!

3 - You may read more and try some of the workarounds listed from the Docker forum, knowingly that at some point in the in the future it might break.

4 - Run Linux instead (or in a VM), which natively routes published container ports to the host without too much of a hassle.