I got a problem with a 1.4.6 Fabric network with a network (27 peers and 5 orderers) where there is one peer (anchor peer) in an Organization that stopped commiting transactions. I can't understand why it started showing this message without any updates on the network, but it was working normally before that.
The message is:
2020-08-26 19:56:35.147 UTC [gossip.privdata] fetchPrivateData -> WARN fc2 Do not know any peer in the channel( xxxx ) that matches the policies , aborting
2020-08-26 19:56:35.147 UTC [gossip.privdata] fetchFromPeers -> WARN fc3 Failed fetching private data for block 743444 from peers: Empty membership
2020-08-26 19:56:36.149 UTC [gossip.privdata] fetchPrivateData -> WARN fc4 Do not know any peer in the channel( xxxx ) that matches the policies , aborting
I already tried to update the chaincode across all peers to see if something changes, but even through all the other peers are updated and still working with their respectives PDCs, this one stopped updating the chaincode as well.
I know we should have other peers configured to disseminate the pvt data, but unfortunately we didn't did it and now i need to find a way to make this peer work again. All the others 26 are fine and they all have the same config (changing only the Organization). Can anyone help me to find a way to just make this peer accept and commit new transaction even if it causes some of the pvt data to be lost ?
Editing for some more info. When i try to send a new transaction for this peer, here's what it happens:
2020-08-28 17:29:07.018 UTC [endorser] callChaincode -> INFO 3c0b [channel][6b3e2fcc] Entry chaincode: name:"chaincode"
2020-08-28 17:29:07.022 UTC [endorser] callChaincode -> INFO 3c0c [channel][6b3e2fcc] Exit chaincode: name:"chaincode" (4ms)
2020-08-28 17:29:07.033 UTC [comm.grpc.server] 1 -> INFO 3c0d unary call completed grpc.service=protos.Endorser grpc.method=ProcessProposal grpc.peer_address=172.21.0.4:40998 grpc.code=OK grpc.call_duration=17.194301ms
2020-08-28 17:29:53.670 UTC [gossip.privdata] StoreBlock -> WARN 3c2f [channel] Could not fetch all missing collection private write sets from remote peers. Will commit block [744876] with missing private write sets:[txID: bdeb55aa80d4c2a2f615abeefe0dbb97a60a08babb5ef2a1f9a0627fe4bf2ccb, seq: 0, namespace: chaincode, collection: collectionTestResults, hash: e0cde0ce12a35de1e7628e9283b26e20849ea5e112ef0daeb8a5a6d7aa1a1706
txID: f505a249ca1c4136711c8402cb2333a2e1b59cb02b573749f4c9488194e3a682, seq: 1, namespace: chaincode, collection: collectionTestResults, hash: ca65f8ed487f7e06201f25a7e0872c522afcb54c1c64c137cec8ef1d31e56d6d
txID: 3032cc2c4ce4702ef1ec0da28ca7e5a0bd4b2c4604915704ae63fc9d8342c138, seq: 2, namespace: chaincode, collection: collectionTestResults, hash: 35c4be3879a191f9e2f132188b313bf4f297e2881e4ad6149c40708191d972fb
]
2020-08-28 17:29:53.675 UTC [statebasedval] ValidateAndPrepareBatch -> WARN 3c30 Block [744876] Transaction index [0] TxId [bdeb55aa80d4c2a2f615abeefe0dbb97a60a08babb5ef2a1f9a0627fe4bf2ccb] marked as invalid by state validator. Reason code [MVCC_READ_CONFLICT]
2020-08-28 17:29:53.675 UTC [statebasedval] ValidateAndPrepareBatch -> WARN 3c31 Block [744876] Transaction index [1] TxId [f505a249ca1c4136711c8402cb2333a2e1b59cb02b573749f4c9488194e3a682] marked as invalid by state validator. Reason code [MVCC_READ_CONFLICT]
2020-08-28 17:29:53.699 UTC [kvledger] CommitWithPvtData -> INFO 3c32 [channel] Committed block [744876] with 3 transaction(s) in 29ms (state_validation=4ms block_and_pvtdata_commit=4ms state_commit=19ms) commitHash=[128eff402fae08d58f50e6529e8e9903116374cac557901bad4fd666153c55aa]
After that, i query the ledger for this specific document, looked inside couchdb, but he's not added to the world state or to the PDC.
Another thing suspicious is that his blocks are way behind the orderer, but he doesn't seems to be fetching them, he's accepting queries normally and even commiting transactions that doesn't use this PDC.
You will get this error if the gossip layer can't find any other peers in the network that have access to this collection. Check the peer logs for the gossip "Membership view" messages. These messages will state which other peers are known by this peer. If you don't see such messages, restart the peer so that you can see in the logs which other peers show up in the new "Membership view" messages.
Typically these issues are related to gossip configuration - double check your configuration values for:
peer.gossip.bootstrap
peer.gossip.endpoint
peer.gossip.externalEndpoint
And make sure the peer can access the bootstrap peer addresses, and that other peers in the org can access this peer via the endpoint address, and that other peers in other orgs can access this peer via the externalEndpoint address.
Once your peer makes connection to another peer that belongs to the same collection, it will reconcile (retrieve) the private data that was missed during this period.