I have ScyllaDB cluster running successfully on docker, but I'm having some connectivity problems when trying to connect my Spring Boot application to it.
The cluster seems to be healthy:
Datacenter: DC1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 172.19.0.3 205.31 KB 256 ? c387bd38-cf37-4c70-a8a3-db586f6410c4 Rack1
UN 172.19.0.2 101.61 KB 256 ? 89dda41e-70c5-43a5-a4a7-ce126007a1e1 Rack1
UN 172.19.0.4 210.34 KB 256 ? 58dfe868-a7ac-47a5-b3e9-dae610ef0bc9 Rack1
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b90d0357abee scylladb/scylla:4.5.0 "/docker-entrypoint.…" 2 minutes ago Up 2 minutes 22/tcp, 7000-7001/tcp, 9042/tcp, 9160/tcp, 9180/tcp, 10000/tcp scylla-node2
19f19cb2e631 scylladb/scylla:4.5.0 "/docker-entrypoint.…" 2 minutes ago Up 2 minutes 22/tcp, 7000-7001/tcp, 9042/tcp, 9160/tcp, 9180/tcp, 10000/tcp scylla-node3
c98b0b1aa018 scylladb/scylla:4.5.0 "/docker-entrypoint.…" 2 minutes ago Up 2 minutes 22/tcp, 7000-7001/tcp, 9042/tcp, 9160/tcp, 9180/tcp, 10000/tcp scylla-node1
And CQL works without any issues:
docker exec -it scylla-node1 cqlsh
Connected to at 172.19.0.2:9042.
[cqlsh 5.0.1 | Cassandra 3.0.8 | CQL spec 3.3.1 | Native protocol v4]
Use HELP for help.
cqlsh>
I'm trying to connect my Java app to the cluster using the datastax drivers (scylla-driver-core version 3.11.2.0):
Cluster cluster = Cluster.builder().addContactPoints("172.19.0.3", "172.19.0.2", "172.19.0.4")
.build();
System.out.println("Connected to cluster " + cluster.getMetadata().getClusterName());
but I'm getting the following exception:
2022-05-08 18:00:02.928 INFO 4760 --- [ restartedMain] com.datastax.driver.core.NettyUtil : Did not find Netty's native epoll transport in the classpath, defaulting to NIO.
2022-05-08 18:00:09.149 WARN 4760 --- [r1-nio-worker-0] com.datastax.driver.core.Connection : Error creating netty channel to /172.19.0.4:9042
io.netty.channel.ConnectTimeoutException: connection timed out: /172.19.0.4:9042
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:261) ~[netty-transport-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) [netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) [netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:164) [netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java) [netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) [netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503) [netty-transport-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.75.Final.jar:4.1.75.Final]
Exception in thread "restartedMain" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.devtools.restart.RestartLauncher.run(RestartLauncher.java:49)
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /172.19.0.4:9042 (com.datastax.driver.core.exceptions.TransportException: [/172.19.0.4:9042] Cannot connect), /172.19.0.3:9042 (com.datastax.driver.core.exceptions.TransportException: [/172.19.0.3:9042] Cannot connect), /172.19.0.2:9042 (com.datastax.driver.core.exceptions.TransportException: [/172.19.0.2:9042] Cannot connect))
at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:270)
at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:109)
at com.datastax.driver.core.Cluster$Manager.negotiateProtocolVersionAndConnect(Cluster.java:1921)
at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1832)
at com.datastax.driver.core.Cluster.getMetadata(Cluster.java:476)
at com.dt.package.ScyllaConnection.connect_and_query(ScyllaConnection.java:15)
at com.dt.package.MyApp.main(MyApp.java:12)
... 5 more
Example of docker-compose file:
scylla-node2:
container_name: scylla-node2
image: scylladb/scylla:4.5.0
restart: always
command: --seeds=scylla-node1,scylla-node2 --smp 1 --memory 750M --overprovisioned 1 --api-address 0.0.0.0
volumes:
- "./scylla/scylla.yaml:/etc/scylla/scylla.yaml"
- "./scylla/cassandra-rackdc.properties.dc1:/etc/scylla/cassandra-rackdc.properties"
networks:
web:
This same code works when trying to connect to a Scylla Cloud cluster (with the proper IPs). Am I missing something simple here?
The problem can be motivated because you are trying to connect to the ScyllaDB nodes with their docker internal IP addresses.
If you need to reach your node from outside of the internal docker network you need to properly expose the appropriate ports to the docker host.
Using docker compose you can export a container port using the ports field. For example:
scylla-node2:
container_name: scylla-node2
image: scylladb/scylla:4.5.0
restart: always
command: --seeds=scylla-node1,scylla-node2 --smp 1 --memory 750M --overprovisioned 1 --api-address 0.0.0.0
ports:
- 9043:9042
volumes:
- "./scylla/scylla.yaml:/etc/scylla/scylla.yaml"
- "./scylla/cassandra-rackdc.properties.dc1:/etc/scylla/cassandra-rackdc.properties"
networks:
web:
In the example, we are exposing the container port 9042 to the host port 9043.
You will need similar configurations to the rest of the nodes: you are using the same host for all of them and as a consequence you need to pick a different port for every node.
scylla-node1:
...
ports:
- 9042:9042
...
scylla-node2:
...
ports:
- 9043:9042
...
scylla-node3:
...
ports:
- 9044:9042
...
The different nodes will share the same host, the one in which the docker containers are running. To connect to them from your Spring application you need something like the following:
Cluster cluster =
Cluster.builder()
.addContactPointsWithPorts(
new InetSocketAddress("127.0.0.1", 9042),
new InetSocketAddress("127.0.0.1", 9043),
new InetSocketAddress("127.0.0.1", 9044)
);
System.out.println("Connected to cluster " + cluster.getMetadata().getClusterName());
Please, note the use of the method addContactPointsWithPorts instead of addContactPoints in the code.