hadoophdfshbaseapache-zookeeper

HBase - hbase:metadata holds info about non existing RegionServer ID - "Master startup cannot progress, in holding-pattern until region onlined."

I cannot start Hbase Master because I am getting this error:

[Thread-18] master.HMaster: hbase:meta,,1.1588230740 
    is NOT online; state={1588230740 state=OPEN, ts=1569328636085, server=regionserver17,16020,1566375930434}; 
    ServerCrashProcedures=true. 
    Master startup cannot progress, in holding-pattern until region onlined.

Hbase Master is active and green but actually it is not started properly since it generates those WARNings in logs and actually I cannot even do the list in Hbase shell because then I get error: ERROR: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing

hbase:meta is referencing some non existing ID 1566375930434 which does not exist in WALs nor in zookeeper-client /hbase-unsecure/rs list.

I tried with these commands:

$ sudo -u hdfs hdfs dfs -rm -r /apps/hbase/data/WALs/

$ zookeeper-client rmr /hbase-unsecure/rs

I also tried and this:

rm -f /var/lib/ambari-metrics-collector/hbase-tmp/zookeeper/zookeeper_0/version-2/*

and restarted the Hbase but still always having the same issue. If anyone can give me additional advice what to try. Thanks

Solution

  • We resolved this issue. Solution is to