Why Does eth.blockNumber Stall at 1 in My Private PoA Clique Ethereum Fork with Geth?

I’m running a private Proof-of-Authority (PoA) Ethereum fork using Geth with the Clique consensus algorithm and a custom chainId (1698369). My setup consists of four servers:

1. Public RPC node behind NGINX.
2. Private RPC + 3 validators (on different ports).
3. Private RPC + 3 validators (on different ports).
4. Private RPC + 3 validators (on different ports).

All 13 services (1 public RPC, 3 private RPCs, 9 validators) start successfully from genesis.json and show as Active in systemctl. However, when querying eth.blockNumber on any RPC or validator node, it remains stuck at 1, even after hours of runtime.

To debug, I inspected logs for one validator with:

journalctl -u geth-mainnet-validator1.service -n 50000 | grep -E "Block|seal|mine|Commit|unlock"

Here’s a sample of the output across validators:

Mar 21 01:10:53 node2 test-geth-mainnet-validator1[139532]: INFO [03-21|01:10:53.493] Commit new sealing work                  number=1 sealhash=d264b8..b8970e txs=0 gas=0 fees=0 elapsed="148.258µs"
Mar 21 01:10:53 node2 test-geth-mainnet-validator1[139532]: WARN [03-21|01:10:53.493] Block sealing failed                     err="unauthorized signer"
Mar 21 01:13:28 node2 test-geth-mainnet-validator1[139532]: INFO [03-21|01:13:28.162] Commit new sealing work                  number=2 sealhash=f12875..f24402 txs=0 gas=0 fees=0 elapsed="115.696µs"
Mar 21 01:13:28 node2 test-geth-mainnet-validator1[139532]: WARN [03-21|01:13:28.162] Block sealing failed                     err="unauthorized signer"
Mar 21 01:11:03 node2 test-geth-mainnet-validator3[139562]: INFO [03-21|01:11:03.659] Successfully sealed new block            number=1 sealhash=bfc8a4..647af4 hash=77a147..1771e5 elapsed=7.713ms
Mar 21 01:11:03 node2 test-geth-mainnet-validator3[139562]: INFO [03-21|01:11:03.660] Commit new sealing work                  number=2 sealhash=1310ab..2ea0d8 txs=0 gas=0 fees=0 elapsed="543.651µs"
Mar 21 01:11:03 node2 test-geth-mainnet-validator3[139562]: WARN [03-21|01:11:03.660] Block sealing failed                     err="signed recently, must wait for others"

The logs show validators attempting to seal blocks, but most fail with err="unauthorized signer", except for one instance where validator3 seals block 1. After that, no further blocks are produced, and I see err="signed recently, must wait for others".

Here’s my genesis.json:

{
  "config": {
    "chainId": 1698369,
    "homesteadBlock": 0,
    "eip150Block": 0,
    "eip155Block": 0,
    "eip158Block": 0,
    "clique": {
      "period": 5,
      "epoch": 30000
    }
  },
  "nonce": "0x0",
  "timestamp": "0x67d1ddfd",
  "extraData": "0x00000000000000000000000000000000000000000000000000000000000000008582102B6e433AEb76B4baA2247c8BB67305405644C41862AFe35E7ffA5d46D106E78e56282106D24C0F541D9e0b6026dcee2532F778dB15E52AA716C17CfBaa87Ec82a26c2Ed9e0206aBF8c74d39d21eDDC1aD264D782A598DeB424d284DA751b0237eE44E3bA47fB9c036e3f9441F8c817d58f0d714c7FC79DE6A1eefAA4325B71590585B4b056B0750e97CEb07760b2b9797b7e31cfd648F7302925c28d582C1EB859B739829ea7D3B99f4445710EfBED2017F7bCeae4a6B59e451f97C6D91BA7115C8ed0c00d68a87bA8cb51aC05422138f639171d36f831F27c0000000000000000000000000000000000000000000000000000000000000000",
  "gasLimit": "0x989680",
  "difficulty": "0x1",
  "mixHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
  "coinbase": "0x0000000000000000000000000000000000000000",
  "alloc": {
    "8582102B6e433AEb76B4baA2247c8BB673054056": {"balance": "180000000000000000000000"},
    "44C41862AFe35E7ffA5d46D106E78e56282106D2": {"balance": "180000000000000000000000"},
    "4C0F541D9e0b6026dcee2532F778dB15E52AA716": {"balance": "180000000000000000000000"},
    "C17CfBaa87Ec82a26c2Ed9e0206aBF8c74d39d21": {"balance": "180000000000000000000000"},
    "eDDC1aD264D782A598DeB424d284DA751b0237eE": {"balance": "180000000000000000000000"},
    "44E3bA47fB9c036e3f9441F8c817d58f0d714c7F": {"balance": "180000000000000000000000"},
    "C79DE6A1eefAA4325B71590585B4b056B0750e97": {"balance": "180000000000000000000000"},
    "CEb07760b2b9797b7e31cfd648F7302925c28d58": {"balance": "180000000000000000000000"},
    "2C1EB859B739829ea7D3B99f4445710EfBED2017": {"balance": "180000000000000000000000"},
    "F7bCeae4a6B59e451f97C6D91BA7115C8ed0c00d": {"balance": "1361000000000000000000000"},
    "d2DEBaecF0591Ab97a4e1e8214fd357390f1879C": {"balance": "0"},
    "68a87bA8cb51aC05422138f639171d36f831F27c": {"balance": "99997019000000000000000000000"}
  }
}

I believe I see a problem.. All your validators are signing genesis.. only one should.. Try this might fix it..

Static Enodes: Copy existing nodekeys from data-mainnet-*/ to a backup dir (e.g., ~/go-ethereum/nodekey-backup/)—to preserve original enodes:

mkdir -p ~/go-ethereum/nodekey-backup cp ~/go-ethereum/data-mainnet-*/nodekey ~/go-ethereum/nodekey-backup/

Service Update: Add --nodekey to all 13 systemd services using the backup keys, used full enode list in --bootnodes:

# Example for validator1 ExecStart=/home/superuser/go-ethereum/build/bin/geth \     --nodekey /home/superuser/go-ethereum/data-mainnet-validator1/nodekey \     --bootnodes "enode://dcdc208...@IP:30303,..."

New genesis: Change you genesis to this (you have an error in the extraData):

{
  "config": {
    "chainId": 698369,
    "homesteadBlock": 0,
    "eip150Block": 0,
    "eip155Block": 0,
    "eip158Block": 0,
    "clique": {
      "period": 5,
      "epoch": 30000
    }
  },
  "nonce": "0x0",
  "timestamp": "0x67dda630",
  "extraData": "0x0000000000000000000000000000000000000000000000000000000000000000c79de6a1eefaa4325b71590585b4b056b0750e97ceb07760b2b9797b7e31cfd648f7302925c28d582c1eb859b739829ea7d3b99f4445710efbed2017f7bceae4a6b59e451f97c6d91ba7115c8ed0c00d68a87ba8cb51ac05422138f639171d36f831f27c4c0f541d9e0b6026dcee2532f778db15e52aa7168582102b6e433aeb76b4baa2247c8bb673054056c17cfbaa87ec82a26c2ed9e0206abf8c74d39d21eddc1ad264d782a598deb424d284da751b0237ee0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
  "gasLimit": "0x989680",
  "difficulty": "0x1",
  "mixHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
  "coinbase": "0x0000000000000000000000000000000000000000",
  "alloc": {
    "c79de6a1eefaa4325b71590585b4b056b0750e97": {"balance": "180000000000000000000000"},
    "ceb07760b2b9797b7e31cfd648f7302925c28d58": {"balance": "180000000000000000000000"},
    "2c1eb859b739829ea7d3b99f4445710efbed2017": {"balance": "180000000000000000000000"},
    "f7bceae4a6b59e451f97c6d91ba7115c8ed0c00d": {"balance": "180000000000000000000000"},
    "68a87ba8cb51ac05422138f639171d36f831f27c": {"balance": "180000000000000000000000"},
    "4c0f541d9e0b6026dcee2532f778db15e52aa716": {"balance": "180000000000000000000000"},
    "8582102b6e433aeb76b4baa2247c8bb673054056": {"balance": "180000000000000000000000"},
    "c17cfbaa87ec82a26c2ed9e0206abf8c74d39d21": {"balance": "180000000000000000000000"},
    "eddc1ad264d782a598deb424d284da751b0237ee": {"balance": "180000000000000000000000"},
    "44e3ba47fb9c036e3f9441f8c817d58f0d714c7f": {"balance": "1361000000000000000000000"},
    "d2debaecf0591ab97a4e1e8214fd357390f1879c": {"balance": "0"},
    "44c41862afe35e7ffa5d46d106e78e56282106d2": {"balance": "99997019000000000000000000000"}
  }
}

Wipe Chain: Cleared geth/ dirs, reinited:

rm -rf ~/go-ethereum/data-mainnet-*/geth for dir in data-mainnet-*; do geth --datadir $dir init genesis.json; done

Staggered Start: Launched validator 1 (60s), then validator 2 (60s), then all 7 others—synced one block 1:

systemctl start test-geth-mainnet-validator1.service; sleep 60 systemctl start test-geth-mainnet-validator2.service; sleep 60 systemctl start test-geth-mainnet-*.service

Sent Tx: Triggered rotation:

geth attach data-mainnet-validator1/geth.ipc > eth.sendTransaction({from: "0xc79d...", to: "0xc79d...", value: web3.toWei(1, "ether"), nonce: 0})

That should help you fix the problem..