I got a network with 3 ORGs and 1 channel. Each ORG has 1 peer and 1 orderer. Whenever I submit a transaction, say peer from org1 tries to submit to orderer from ORG2 and fails, then tries to submit to orderer from ORG3 and fails, and finally tries with orderer from ORG1 and it connects. Same happens for the other peers, the TX are successfully submitted only when they contact the orderer from their own ORG. That seems correct to me, I think the problem is the peer trying to submit to an orderer from a different ORG. Is this a channel setting that I should review? Because as I said, I think it's working correctly, but the peers shouldn't be tryng to reach an orderer from a different ORG.
WARN [gateway] orderers -> Failed to connect to orderer address=orderer.org2... err="failed to create new connection: connection error: desc = \"transport: error while dialing: dial tcp ...: connect: connection refused\""
WARN [gateway] orderers -> Failed to connect to orderer address=orderer.org3... err="failed to create new connection: connection error: desc = \"transport: error while dialing: dial tcp ...: connect: connection refused\""
INFO [gateway] Submit -> Sending transaction to orderer txID=... endpoint=orderer.org1.
Some additional information. The orderer are consenters in the channel:
"ConsensusType": {
"mod_policy": "Admins",
"value": {
"metadata": {
"consenters": [
{
"client_tls_cert": "updated cert",
"host": "orderer.org1.xxx.com",
"port": 7050,
"server_tls_cert": "updated cert"
},
{
"client_tls_cert": "updated cert",
"host": "orderer.org2.x.com",
"port": 8050,
"server_tls_cert": "updated cert"
},
{
"client_tls_cert": "old cert",
"host": "orderer.org3.x.com",
"port": 9050,
"server_tls_cert": "old cert"
}
],
"options": {
"election_tick": 10,
"heartbeat_tick": 1,
"max_inflight_blocks": 5,
"snapshot_interval_size": 16777216,
"tick_interval": "500ms"
}
},
"OrdererAddresses": {
"mod_policy": "/Channel/Orderer/Admins",
"value": {
"addresses": [
"orderer.org1.x.com:7050",
"orderer.org2.x.com:8050",
"orderer.org3.x.com:9050"
]
},
"version": "1"
}
I've updated the certs via channel updates, since when I looked at the logs on an orderer, say orderer3, I noticed that the certs displayed from the other orderers were the old ones. However, after updated certs from org1, and org2, (one by one as stated in docs), now orderer 2 is not able to participate from consensus:
3-10-15 23:30:38.272 UTC 0089 INFO [orderer.consensus.etcdraft] Step -> 3 [logterm: 14, index: 94826, vote: 3] ignored MsgPreVote from 2 [logterm: 14, index: 94823] at term 14: lease is not expired (remaining ticks: 2) channel=channel1 node=3
2023-10-15 23:30:44.269 UTC 008a INFO [orderer.consensus.etcdraft] Step -> 3 [logterm: 14, index: 94826, vote: 3] ignored MsgPreVote from 2 [logterm: 14, index: 94823] at term 14: lease is not expired (remaining ticks: 10) channel=channel1 node=3
3-10-15 23:30:50.269 UTC 008b INFO [orderer.consensus.etcdraft] Step -> 3 [logterm: 14, index: 94826, vote: 3] ignored MsgPreVote from 2 [logterm: 14, index: 94823] at term 14: lease is not expired (remaining ticks: 8) channel=channel1 node=3
3-10-15 23:30:56.269 UTC 008c INFO [orderer.consensus.etcdraft] Step -> 3 [logterm: 14, index: 94826, vote: 3] ignored MsgPreVote from 2 [logterm: 14, index: 94823] at term 14: lease is not expired (remaining ticks: 6) channel=channel1 node=3
3-10-15 23:31:02.269 UTC 008d INFO [orderer.consensus.etcdraft] Step -> 3 [logterm: 14, index: 94826, vote: 3] ignored MsgPreVote from 2 [logterm: 14, index: 94823] at term 14: lease is not expired (remaining ticks: 4) channel=channel1 node=3
3-10-15 23:31:08.269 UTC 008e INFO [orderer.consensus.etcdraft] Step -> 3 [logterm: 14, index: 94826, vote: 3] ignored MsgPreVote from 2 [logterm: 14, index: 94823] at term 14: lease is not expired (remaining ticks: 2) channel=channel1 node=3
This is log from orderer3 for reference:
3-10-13 18:50:11.971 UTC 0011 INFO [orderer.commmon.multichannel] initSystemChannel -> Starting system channel 'system-channel' with genesis block hash c9e6204d3e77735704a77cf3055be51997a4abe452e7d9b575caf0f023a82325 and orderer type etcdraft
2023-10-13 18:50:11.985 UTC 0012 INFO [orderer.consensus.etcdraft] HandleChain -> EvictionSuspicion not set, defaulting to 10m0s
2023-10-13 18:50:11.985 UTC 0013 INFO [orderer.consensus.etcdraft] HandleChain -> With system channel: after eviction InactiveChainRegistry.TrackChain will be called
2023-10-13 18:50:11.985 UTC 0014 INFO [orderer.consensus.etcdraft] createOrReadWAL -> Found WAL data at path '/var/hyperledger/production/orderer/etcdraft/wal/channel1', replaying it channel=channel1 node=3
2023-10-13 18:50:12.132 UTC 0015 INFO [orderer.consensus.etcdraft] Start -> Starting Raft node channel=system-channel node=3
2023-10-13 18:50:12.132 UTC 0016 INFO [orderer.common.cluster] Configure -> Entering, channel: system-channel, nodes: [ID: 1,
Endpoint: orderer.org1.xxx.com:7050,
ServerTLSCert:-----BEGIN CERTIFICATE-----
OLD CERT
-----END CERTIFICATE-----
, ClientTLSCert:-----BEGIN CERTIFICATE-----
OLD CERT
-----END CERTIFICATE-----
ID: 2,
Endpoint: orderer.org2.xxx.com:7050,
ServerTLSCert:-----BEGIN CERTIFICATE-----
OLD CERT
-----END CERTIFICATE-----
, ClientTLSCert:-----BEGIN CERTIFICATE-----
OLD CERT
-----END CERTIFICATE-----
]
2023-10-13 18:50:12.132 UTC 0017 INFO [orderer.common.cluster] updateStubInMapping -> Allocating a new stub for node 1 with endpoint of orderer.org1.xxx.com:7050 for channel system-channel
2023-10-13 18:50:12.132 UTC 0018 INFO [orderer.common.cluster] updateStubInMapping -> Deactivating node 1 in channel system-channel with endpoint of orderer.org1.xxx.com:7050 due to TLS certificate change
2023-10-13 18:50:12.133 UTC 0019 INFO [orderer.common.cluster] updateStubInMapping -> Allocating a new stub for node 2 with endpoint of orderer.org2.xxx.com:7050 for channel system-channel
2023-10-13 18:50:12.133 UTC 001a INFO [orderer.common.cluster] updateStubInMapping -> Deactivating node 2 in channel system-channel with endpoint of orderer.org2.xxx.com:7050 due to TLS certificate change
2023-10-13 18:50:12.134 UTC 001b INFO [orderer.common.cluster] func1 -> 1 exists in both old and new membership for channel system-channel , skipping its deactivation
2023-10-13 18:50:12.134 UTC 001c INFO [orderer.common.cluster] func1 -> 2 exists in both old and new membership for channel system-channel , skipping its deactivation
2023-10-13 18:50:12.134 UTC 001d INFO [orderer.common.cluster] Configure -> Exiting
2023-10-13 18:50:12.134 UTC 001e INFO [orderer.consensus.etcdraft] start -> Restarting raft node channel=system-channel node=3
2023-10-13 18:50:12.135 UTC 001f INFO [orderer.consensus.etcdraft] becomeFollower -> 3 became follower at term 9 channel=system-channel node=3
2023-10-13 18:50:12.135 UTC 0020 INFO [orderer.consensus.etcdraft] newRaft -> newRaft 3 [peers: [], term: 9, commit: 11, applied: 0, lastindex: 12, lastterm: 9] channel=system-channel node=3
2023-10-13 18:50:12.135 UTC 0021 INFO [orderer.consensus.etcdraft] Start -> Starting Raft node channel=channel1 node=3
2023-10-13 18:50:12.135 UTC 0022 INFO [orderer.common.cluster] Configure -> Entering, channel: channel1, nodes: [ID: 1,
Endpoint: orderer.org1.xxx.com:7050,
ServerTLSCert:-----BEGIN CERTIFICATE-----
NEW CERT
-----END CERTIFICATE-----
, ClientTLSCert:-----BEGIN CERTIFICATE-----
NEW CERT
-----END CERTIFICATE-----
ID: 2,
Endpoint: orderer.org2.xxx.com:8050,
ServerTLSCert:-----BEGIN CERTIFICATE-----
NEW CERT
-----END CERTIFICATE-----
, ClientTLSCert:-----BEGIN CERTIFICATE-----
NEW CERT
-----END CERTIFICATE-----
]
2023-10-13 18:50:12.135 UTC 0023 INFO [orderer.common.cluster] updateStubInMapping -> Allocating a new stub for node 1 with endpoint of orderer.org1.xxx.com:7050 for channel channel1
2023-10-13 18:50:12.135 UTC 0024 INFO [orderer.common.cluster] updateStubInMapping -> Deactivating node 1 in channel channel1 with endpoint of orderer.org1.xxx.com:7050 due to TLS certificate change
2023-10-13 18:50:12.135 UTC 0025 INFO [orderer.common.cluster] updateStubInMapping -> Allocating a new stub for node 2 with endpoint of orderer.org2.xxx.com:8050 for channel channel1
2023-10-13 18:50:12.135 UTC 0026 INFO [orderer.common.cluster] updateStubInMapping -> Deactivating node 2 in channel channel1 with endpoint of orderer.org2.xxx.com:8050 due to TLS certificate change
2023-10-13 18:50:12.136 UTC 0027 INFO [orderer.common.cluster] func1 -> 1 exists in both old and new membership for channel channel1 , skipping its deactivation
2023-10-13 18:50:12.136 UTC 0028 INFO [orderer.common.cluster] func1 -> 2 exists in both old and new membership for channel channel1 , skipping its deactivation
2023-10-13 18:50:12.136 UTC 0029 INFO [orderer.common.cluster] Configure -> Exiting
2023-10-13 18:50:12.136 UTC 002a INFO [orderer.consensus.etcdraft] start -> Restarting raft node channel=channel1 node=3
2023-10-13 18:50:12.136 UTC 002b INFO [orderer.consensus.etcdraft] becomeFollower -> 3 became follower at term 10 channel=channel1 node=3
2023-10-13 18:50:12.136 UTC 002c INFO [orderer.consensus.etcdraft] newRaft -> newRaft 3 [peers: [1,2,3], term: 10, commit: 94811, applied: 93447, lastindex: 94812, lastterm: 10] channel=channel1 node=3
2023-10-13 18:50:12.137 UTC 002d INFO [orderer.common.server] Main -> Starting orderer:
Version: 2.4.9
Commit SHA: 2a13518
Go version: go1.18.10
OS/Arch: linux/amd64
2023-10-13 18:50:12.137 UTC 002e INFO [orderer.common.server] Main -> Beginning to serve requests
The peer Gateway service intentionally randomises the sequence of available orderers to which it submits transactions on each invocation to balance load across orderers. All orderers should be accessible to all network peers so that they can:
The issue appears to be that your peers are not able to connect to orderers outside their organisation. Perhaps this is a network configuration / firewall issue?