node.jscassandrakubernetescqlshdatastax-node-driver

Expose Cassandra running on Kubernetes

I am running Cassandra on Kubernetes (3 instances) and want to expose it to the outside, my application is not yet in Kubernetes. So i crated a load balanced service like so:

apiVersion: v1
kind: Service
metadata:
  namespace: getquanty
  labels:
    app: cassandra
  name: cassandra
  annotations:
    kubernetes.io/tls-acme: "true"
spec:
  clusterIP:
  ports:
    - port: 9042
      name: cql
      nodePort: 30001
    - port: 7000
      name: intra-node
      nodePort: 30002
    - port: 7001
      name: tls-intra-node
      nodePort: 30003
    - port: 7199
      name: jmx
      nodePort: 30004
  selector:
    app: cassandra
  type: LoadBalancer

This is the result is:

NAME        CLUSTER-IP     EXTERNAL-IP     PORT(S)                                                       AGE
cassandra   10.55.249.88   GIVEN_IP_GCE_LB   9042:30001/TCP,7000:30002/TCP,7001:30003/TCP,7199:30004/TCP   26m

I am able to connect using sh (cqlsh GIVEN_IP_GCE_LB ) but when i try to add data to Cassandra using the datastax driver for node, i got this:

message: 'Cannot achieve consistency level SERIAL',
       info: 'Represents an error message from the server',
       code: 4096,
       consistencies: 8,
       required: 1,
       alive: 0,
       coordinator: '35.187.166.68:9042' },
    '10.52.4.32:9042': 'Host considered as DOWN',
    '10.52.2.15:9042': 'Host considered as DOWN' },
 info: 'Represents an error when a query cannot be performed because no host is available or could be reached by the driver.',
 message: 'All host(s) tried for query failed. First host tried, 35.187.166.68:9042: ResponseError: Cannot achieve consistency level SERIAL. See innerErrors.' }

My first though was I need to expose the other ports too, so I did (intra-node, tls-intra-node, jmx), but it was the same error.

Kubernetes gives you access to proxy, i tried to proxy from my machine using the constructed URL for the pod to test if i have access but i cannot connect using cqlsh:

http://127.0.0.1:8001/api/v1/namespaces/qq/pods/cassandra-0:cql/proxy

I am out of ideas, the one thing left to try is to expose every instance (make a service for every instance) which is very ugly, but it will let me connect to the nodes from the outside until i migrate the application to Kubernetes.

Does any one have ideas how to expose Cassandra nodes to the internet and make the Datastax driver aware of all the nodes? Thank you for your time.

Solution

  • After more reading I found out that the replication strategy was the one causing the problem, NetworkStrategy is suitable for multi-cluster, I have one, so I changed the replication to simple with the number of nodes i had, now every thing works as expected.

    EDIT 1: Putting databases on Kube is not a good solution, I ended up making a standalone cluster, added it to the same Network as kube, and was able to access it from kube pods.

    Kube is made to manage application and make them 'elastic', i don't think people really need to scale databases as quick as applications, furthermore, the scaling of a database is not the same operation as a stateless application.