apache-zookeepercloudera-quickstart-vm

Zookeeper running or not in relation to standard port 2181 usage?


CLOUDERA QUICKSTART 5.13 as follows.

I am not sure whether zookeeper out of the box is running or not, and if so, then if it would work reliably? I got this when trying to run zookeeper from within the from kafka supplied version that I downloaded, in standalone mode:

[2018-06-17 00:49:32,847] INFO binding to port 0.0.0.0/0.0.0.0:2181 
(org.apache.zookeeper.server.NIOServerCnxnFactory)
[2018-06-17 00:49:32,854] ERROR Unexpected exception, exiting abnormally 
(org.apache.zookeeper.server.ZooKeeperServerMain)
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)

netstat on the vm reveals:

[cloudera@quickstart kafka_2.11-1.1.0]$ netstat -an | grep 2181
tcp        0      0 0.0.0.0:2181                0.0.0.0:*                   
LISTEN      
tcp        0      0 127.0.0.1:2181              127.0.0.1:49718             
ESTABLISHED 
tcp        0      0 127.0.0.1:49707             127.0.0.1:2181              
ESTABLISHED 
tcp        0      0 127.0.0.1:2181              127.0.0.1:49707             
ESTABLISHED 
tcp        0      0 127.0.0.1:49697             127.0.0.1:2181              
ESTABLISHED 
tcp        0      0 10.0.2.15:49065             10.0.2.15:2181              
ESTABLISHED 
tcp        0      0 127.0.0.1:49718             127.0.0.1:2181              
ESTABLISHED  
tcp        0      0 127.0.0.1:49706             127.0.0.1:2181              
ESTABLISHED 
tcp        0      0 127.0.0.1:49714             127.0.0.1:2181              
ESTABLISHED 
tcp        0      0 10.0.2.15:2181              10.0.2.15:49060             
ESTABLISHED 
tcp        0      0 10.0.2.15:2181              10.0.2.15:49065             
ESTABLISHED 
tcp        0      0 127.0.0.1:2181              127.0.0.1:49701             
ESTABLISHED 
tcp        0      0 127.0.0.1:2181              127.0.0.1:49714             
ESTABLISHED 
tcp        0      0 127.0.0.1:2181              127.0.0.1:49706             
ESTABLISHED 
tcp        0      0 10.0.2.15:49060             10.0.2.15:2181              
ESTABLISHED 
tcp        0      0 127.0.0.1:49701             127.0.0.1:2181              
ESTABLISHED 
tcp        0      0 127.0.0.1:2181              127.0.0.1:49697             
ESTABLISHED 

sudo jps when executed shows QuorumPeerMain - which I think is zookeeper these days(?):

 8196 
 5559 SecondaryNameNode
 7116 HistoryServer
 5831 NodeManager
 5290 DataNode
 10995 Jps
 5216 QuorumPeerMain
 6449 ThriftServer
 6587 RunJar
 7068 Bootstrap
 5384 JournalNode
 7879 Bootstrap
 6317 RESTServer
 7237 HRegionServer
 5687 Bootstrap
 6061 ResourceManager
 8124 Bootstrap
 8153 
 5479 NameNode
 5745 JobHistoryServer
 6699 RunJar
 6158 HMaster

Not sure what to make of it as got the below when starting zookeeper from cloudera install. Do I have zookeeper working? No such process to kill means?

[cloudera@quickstart kafka_2.11-1.1.0]$ sudo  
 /usr/lib/zookeeper/bin/zkServer.sh start
JMX enabled by default
Using config: /usr/lib/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[cloudera@quickstart kafka_2.11-1.1.0]$ sudo    /usr/lib/zookeeper/bin/zkServer.sh stop
JMX enabled by default
Using config: /usr/lib/zookeeper/bin/../conf/zoo.cfg
Stopping zookeeper ... /usr/lib/zookeeper/bin/zkServer.sh: line 162: kill: 
(11140) - No such process
STOPPED

Basically, I find the observations not adding up with standard descriptions.


Solution

  • As your error states that you have an

    Address already in use

    you should figure out (as beny23 said) which process occupies that port:

    lsof -i :2181
    

    Once you have a zookeeper running you should see logs like:

    Validating environment
    ZK_REPLICAS=1
    MY_ID=1
    ZK_LOG_LEVEL=INFO
    ZK_DATA_DIR=/var/lib/zookeeper/data
    ZK_DATA_LOG_DIR=/var/lib/zookeeper/log
    ZK_LOG_DIR=/var/log/zookeeper
    ZK_CLIENT_PORT=2181
    ZK_SERVER_PORT=2888
    ZK_ELECTION_PORT=3888
    ZK_TICK_TIME=2000
    ZK_INIT_LIMIT=10
    ZK_SYNC_LIMIT=2000
    ZK_MAX_CLIENT_CNXNS=60
    ZK_MIN_SESSION_TIMEOUT= 4000
    ZK_MAX_SESSION_TIMEOUT= 40000
    ZK_HEAP_SIZE=1G
    ZK_SNAP_RETAIN_COUNT=3
    ZK_PURGE_INTERVAL=0
    ENSEMBLE
    server.1=zookeeper-0.zookeeper.default.svc.cluster.local:2888:3888
    Environment validation successful
    Creating ZooKeeper configuration
    Wrote ZooKeeper configuration file to /etc/zookeeper/zoo.cfg
    Creating ZooKeeper log4j configuration
    Wrote log4j configuration to /etc/zookeeper/log4j.properties
    Creating ZooKeeper data directories and setting permissions
    Created ZooKeeper data directories and set permissions in /var/lib/zookeeper/data
    Creating JVM configuration file
    Wrote JVM configuration to /etc/zookeeper/java.env
    ZooKeeper JMX enabled by default
    Using config: /etc/zookeeper/zoo.cfg
    2018-06-18 07:52:43,747 [myid:] - INFO  [main:QuorumPeerConfig@136] - Reading configuration from: /etc/zookeeper/zoo.cfg
    2018-06-18 07:52:43,752 [myid:] - INFO  [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
    2018-06-18 07:52:43,752 [myid:] - INFO  [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
    2018-06-18 07:52:43,752 [myid:] - INFO  [main:DatadirCleanupManager@101] - Purge task is not scheduled.
    2018-06-18 07:52:43,752 [myid:] - WARN  [main:QuorumPeerMain@116] - Either no config or no quorum defined in config, running  in standalone mode
    2018-06-18 07:52:43,764 [myid:] - INFO  [main:QuorumPeerConfig@136] - Reading configuration from: /etc/zookeeper/zoo.cfg
    2018-06-18 07:52:43,764 [myid:] - INFO  [main:ZooKeeperServerMain@98] - Starting server
    2018-06-18 07:52:43,771 [myid:] - INFO  [main:Environment@100] - Server environment:zookeeper.version=3.4.11-37e277162d567b55a07d1755f0b31c32e93c01a0, built on 11/01/2017 18:06 GMT
    2018-06-18 07:52:43,771 [myid:] - INFO  [main:Environment@100] - Server environment:host.name=zookeeper-0.zookeeper.default.svc.cluster.local
    2018-06-18 07:52:43,771 [myid:] - INFO  [main:Environment@100] - Server environment:java.version=1.8.0_151
    2018-06-18 07:52:43,772 [myid:] - INFO  [main:Environment@100] - Server environment:java.vendor=Oracle Corporation
    2018-06-18 07:52:43,772 [myid:] - INFO  [main:Environment@100] - Server environment:java.home=/usr/lib/jvm/java-8-openjdk-amd64/jre
    2018-06-18 07:52:43,772 [myid:] - INFO  [main:Environment@100] - Server environment:java.class.path=/usr/bin/../build/classes:/usr/bin/../build/lib/*.jar:/usr/bin/../share/zookeeper/zookeeper-3.4.11.jar:/usr/bin/../share/zookeeper/slf4j-log4j12-1.6.1.jar:/usr/bin/../share/zookeeper/slf4j-api-1.6.1.jar:/usr/bin/../share/zookeeper/netty-3.10.5.Final.jar:/usr/bin/../share/zookeeper/log4j-1.2.16.jar:/usr/bin/../share/zookeeper/jline-0.9.94.jar:/usr/bin/../share/zookeeper/audience-annotations-0.5.0.jar:/usr/bin/../src/java/lib/*.jar:/etc/zookeeper:
    2018-06-18 07:52:43,772 [myid:] - INFO  [main:Environment@100] - Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
    2018-06-18 07:52:43,772 [myid:] - INFO  [main:Environment@100] - Server environment:java.io.tmpdir=/tmp
    2018-06-18 07:52:43,772 [myid:] - INFO  [main:Environment@100] - Server environment:java.compiler=<NA>
    2018-06-18 07:52:43,773 [myid:] - INFO  [main:Environment@100] - Server environment:os.name=Linux
    2018-06-18 07:52:43,773 [myid:] - INFO  [main:Environment@100] - Server environment:os.arch=amd64
    2018-06-18 07:52:43,773 [myid:] - INFO  [main:Environment@100] - Server environment:os.version=4.9.64
    2018-06-18 07:52:43,773 [myid:] - INFO  [main:Environment@100] - Server environment:user.name=zookeeper
    2018-06-18 07:52:43,774 [myid:] - INFO  [main:Environment@100] - Server environment:user.home=/home/zookeeper
    2018-06-18 07:52:43,774 [myid:] - INFO  [main:Environment@100] - Server environment:user.dir=/usr/bin
    2018-06-18 07:52:43,778 [myid:] - INFO  [main:ZooKeeperServer@825] - tickTime set to 2000
    2018-06-18 07:52:43,778 [myid:] - INFO  [main:ZooKeeperServer@834] - minSessionTimeout set to 4000
    2018-06-18 07:52:43,778 [myid:] - INFO  [main:ZooKeeperServer@843] - maxSessionTimeout set to 40000
    2018-06-18 07:52:43,785 [myid:] - INFO  [main:ServerCnxnFactory@117] - Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory
    2018-06-18 07:52:43,788 [myid:] - INFO  [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:2181
    2018-06-18 07:52:56,088 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@215] - Accepted socket connection from /127.0.0.1:40634
    2018-06-18 07:52:56,093 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ServerCnxn@324] - The list of known four letter word commands is : [{1936881266=srvr, 1937006964=stat, 2003003491=wchc, 1685417328=dump, 1668445044=crst, 1936880500=srst, 1701738089=envi, 1668247142=conf, 2003003507=wchs, 2003003504=wchp, 1668247155=cons, 1835955314=mntr, 1769173615=isro, 1920298859=ruok, 1735683435=gtmk, 1937010027=stmk}]
    

    In your config you should have define the server name, I have this line (I deployed zookeeper in a kubernetes cluster)

    server.1=zookeeper-0.zookeeper.default.svc.cluster.local:2888:3888
    

    Anyway you need to declare it (in your zoo.cfg) or pass it as variable at startup.