apache-kafkamesosapache-zookeeper

Issue using same zookeeper for Kafka and Mesos

I am trying to setup Kafka and Spark with Mesos on our 8 nodes cluster as following but having issues launching/starting Mesos Agent using zookeeper endpoint of Mesos masters.

  1. Install and setup Zookeeper on 3 nodes (server00,server01,server02) (through $KAFKA_HOME/config/zookeeper.properties)

  2. Install Kafka brokers on all 8 nodes (and point it to 3 zookeepers by setting following property in its $KAFKA_HOME/config/server.properties)

    zookeeper.connect=server00:2181,server01:2181,server02:2181

  3. Install Mesos master on 3 nodes (server00,server01,server02) and update /etc/mesos/zk with following line:

    zk://server00:2181,server01:2181,server02:2181/mesos

  4. Install Mesos agents on all 8 nodes.

  5. Edit /etc/mesos/zk file on all other servers to have following line.

    zk://server00:2181,server01:2181,server02:2181/mesos

  6. Start Mesos master on all 3 master servers as below (verified that all Mesos master are running and available by launching http://server00:5050/#/, http://server01:5050/#/, http://server02:5050/#/

    sudo /usr/sbin/mesos-master --cluster=server_mesos_cluster --log_dir=/var/log/mesos --work_dir=/var/lib/mesos

  7. Start Mesos Agent on all 8 servers. Example of launching this on server00:

sudo /usr/sbin/mesos-slave --work_dir=/var/lib/mesos --master=zk://server00:2181,server01:2181,server02:2181/mesos --ip=9.1.69.150 But above doesn't launch agent.

But following command does which makes me think that perhaps master mesos are not getting registered with zookeepers.

sudo /usr/sbin/mesos-slave --work_dir=/var/lib/mesos --master=server00:5050 --ip=9.1.69.150

Could anyone shed any light as to whether

  1. My configuration is not right or
  2. If I have to setup separate zookeepers for Mesos cluster?
  3. How can I verify if Mesos masters are getting registered with zookeeper?

Once this setup is working, I intend to run Spark on all 8 nodes.

Solution

  • On Ubuntu, at least, /etc/mesos/zk, and other config files under /etc/mesos are only read by /usr/bin/mesos-init-wrapper. Thus your master isn't seeing your zk config.

    You'll either need to launch it with the init script (service mesos-master start), run the wrapper manually, or use the -zk option to mesos-master:

    sudo /usr/sbin/mesos-master --cluster=server_mesos_cluster --log_dir=/var/log/mesos --work_dir=/var/lib/mesos --zk=zk://server00:2181,server01:2181,server02:2181/mesos

    `