My cluster was previously running, I have made these config changes and now Galera cluster bootstrap is not working.
I tried bootstraping with /usr/bin/mysqld_bootstrap as well as service mysql start --wsrep-new-cluster . but no luck.
mysql> SHOW GLOBAL STATUS WHERE Variable_name IN ('wsrep_ready', 'wsrep_cluster_size', 'wsrep_cluster_status', 'wsrep_connected');
+----------------------+--------------+
| Variable_name | Value |
+----------------------+--------------+
| wsrep_cluster_size | 0 |
| wsrep_cluster_status | Disconnected |
| wsrep_connected | OFF |
| wsrep_ready | ON |
+----------------------+--------------+
4 rows in set (0.00 sec)
root@hostname:/var/lib/mysql# cat /etc/my.cnf
[mysqld]
# character set
character_set_server = utf8
collation_server = utf8_general_ci
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
binlog_format=ROW
# Galera Cluster Configuration
wsrep_node_address="ip"
wsrep_node_name="hostname"
wsrep_cluster_name="mysql_cluster"
wsrep_cluster_address="gcomm://"
wsrep_provider_options = "gcache.size = 16G; gcache.recover = yes; gmcast.segment=0; gmcast.time_wait=PT5S; cert.log_conflicts=ON;"
wsrep_retry_autocommit=5
wsrep_slave_threads=64
wsrep_log_conflicts=ON
wsrep_provider=/usr/lib/libgalera_smm.so
wsrep_sst_method=rsync # Galera Synchronization Configuration
# wsrep_certify_nonPK=1 # make sure primary key is present in a DB
# Mysql configurations
default_storage_engine=innodb
innodb_autoinc_lock_mode=2
innodb_flush_log_at_trx_commit=2
innodb_buffer_pool_size=20G # 80% of total memory
table_open_cache=8192
max_connections=500
innodb_buffer_pool_instances=40
thread_cache_size=128
max_allowed_packet=16M
thread_stack=192K
[mysql_safe]
log-error=/var/log/mysql/error.log
pid-file=/var/run/mysqld/mysqld.pid
#include mysql config
!includedir /etc/mysql/conf.d/
error.log
2020-05-10T14:52:11.510387Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2020-05-10T14:52:11.512408Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.29-log) starting as process 38762 ...
2020-05-10T14:52:11.516208Z 0 [Note] InnoDB: PUNCH HOLE support available
2020-05-10T14:52:11.516233Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2020-05-10T14:52:11.516240Z 0 [Note] InnoDB: Uses event mutexes
2020-05-10T14:52:11.516246Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2020-05-10T14:52:11.516251Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2020-05-10T14:52:11.516259Z 0 [Note] InnoDB: Using Linux native AIO
2020-05-10T14:52:11.516566Z 0 [Note] InnoDB: Number of pools: 1
2020-05-10T14:52:11.516689Z 0 [Note] InnoDB: Using CPU crc32 instructions
2020-05-10T14:52:11.518701Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2020-05-10T14:52:11.528085Z 0 [Note] InnoDB: Completed initialization of buffer pool
2020-05-10T14:52:11.530536Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2020-05-10T14:52:11.542537Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2020-05-10T14:52:11.559072Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2020-05-10T14:52:11.559138Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2020-05-10T14:52:11.601120Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2020-05-10T14:52:11.602357Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
2020-05-10T14:52:11.602377Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
2020-05-10T14:52:11.602854Z 0 [Note] InnoDB: Waiting for purge to start
2020-05-10T14:52:11.653077Z 0 [Note] InnoDB: 5.7.29 started; log sequence number 3779657135
2020-05-10T14:52:11.653388Z 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2020-05-10T14:52:11.653670Z 0 [Note] Plugin 'FEDERATED' is disabled.
2020-05-10T14:52:11.656614Z 0 [Note] InnoDB: Buffer pool(s) load completed at 200510 7:52:11
2020-05-10T14:52:11.665175Z 0 [Note] Found ca.pem, server-cert.pem and server-key.pem in data directory. Trying to enable SSL support using them.
2020-05-10T14:52:11.665203Z 0 [Note] Skipping generation of SSL certificates as certificate files are present in data directory.
2020-05-10T14:52:11.666274Z 0 [Warning] CA certificate ca.pem is self signed.
2020-05-10T14:52:11.666344Z 0 [Note] Skipping generation of RSA key pair as key files are present in data directory.
2020-05-10T14:52:11.667341Z 0 [Note] Server hostname (bind-address): '*'; port: 3306
2020-05-10T14:52:11.667427Z 0 [Note] IPv6 is available.
2020-05-10T14:52:11.667444Z 0 [Note] - '::' resolves to '::';
2020-05-10T14:52:11.667487Z 0 [Note] Server socket created on IP: '::'.
2020-05-10T14:52:11.686718Z 0 [Note] Event Scheduler: Loaded 0 events
2020-05-10T14:52:11.686771Z 0 [Note] WSREP: Initial position: 00000000-0000-0000-b0f1-bf0300000000:62901600
2020-05-10T14:52:11.686784Z 0 [Warning] WSREP: Initial position was provided by configuration or SST, avoiding override
2020-05-10T14:52:11.686792Z 0 [Note] WSREP: wsrep_load(): loading provider library 'none'
2020-05-10T14:52:11.687082Z 0 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.7.29-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 MySQL Wsrep Server (GPL), wsrep_25.21
2020-05-10T14:52:12.732431Z 2 [Note] Access denied for user 'root'@'localhost' (using password: NO)
2020-05-10T14:57:11.957879Z 4 [Note] WSREP: Stop replication
2020-05-10T14:57:11.957918Z 4 [Note] Giving 1 client threads a chance to die gracefully
The file /etc/my.cnf was missing read permissions. This file requires read permission for mysql user. I have set the permission to 755 and Galera configuration was loaded properly.
chmod 775 /etc/my.cnf