Failed to connect before the deadline on Endorser

connection-profile.yaml

{
    "name": "network",
    "version": "1.0.0",
    "client": {
        "organization": "Freight",
        "connection": {
            "timeout": {
                "peer": {
                    "endorser": "300"
                }
            }
        }
    },
    "organizations": {
        "Freight": {
            "mspid": "FreightMSP",
            "peers": [
                "peer0.freight.example.com",
                "peer1.freight.example.com"
            ],
            "certificateAuthorities": [
                "ca.freight.example.com"
            ]
        }
    },
    "peers": {
        "peer0.freight.example.com":{
            "url": "grpcs://peer0.freight.example.com:9051",
            "grpcOption": {
                "ssl-target-name-override": "peer0.freight.example.com",
                "hostnameOverride": "peer0.freight.example.com"
            },
            "tlsCACerts": {
                "path": "../../network/organizations/peerOrganizations/freight.example.com/tlsca/tlsca.freight.example.com-cert.pem"
            }
        },
        "peer1.freight.example.com":{
            "url": "grpcs://peer1.freight.example.com:10051",
            "grpcOption": {
                "ssl-target-name-override": "peer1.freight.example.com",
                "hostnameOverride": "peer1.freight.example.com"
            },
            "tlsCACerts": {
                "path": "../../network/organizations/peerOrganizations/freight.example.com/tlsca/tlsca.freight.example.com-cert.pem"
            }
        }
    },
    "certificateAuthorities": {
        "ca.freight.example.com": {
            "url": "https://ca.freight.example.com:9054",
            "caName": "ca.freight.example.com",
            "httpOptions": {
                "verify": false
            },
            "tlsCACerts": {
                "path": "../../network/organizations/peerOrganizations/freight.example.com/ca/ca.freight.example.com-cert.pem"
            }
        }
    }
}

Discovery Service:

discovery: { 
        enabled: true, 
        asLocalhost: false 
    } 

asLocalhost is false because all orgs are hosted on different hosts and are in docker swarm network. But i get this error when call the add asset invoke function.

    Wallet Path: /home/hlfabric/fabric/api/freight/wallet
2021-03-06T17:58:15.407Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Endorser- name: peer0.freight.example.com, url:grpcs://peer0.freight.example.com:9051, connected:false, connectAttempted:true
2021-03-06T17:58:15.408Z - error: [ServiceEndpoint]: waitForReady - Failed to connect to remote gRPC server peer0.freight.example.com url:grpcs://peer0.freight.example.com:9051 timeout:3000
2021-03-06T17:58:15.409Z - info: [NetworkConfig]: buildPeer - Unable to connect to the endorser peer0.freight.example.com due to Error: Failed to connect before the deadline on Endorser- name: peer0.freight.example.com, url:grpcs://peer0.freight.example.com:9051, connected:false, connectAttempted:true
2021-03-06T17:58:18.412Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Endorser- name: peer1.freight.example.com, url:grpcs://peer1.freight.example.com:10051, connected:false, connectAttempted:true
2021-03-06T17:58:18.412Z - error: [ServiceEndpoint]: waitForReady - Failed to connect to remote gRPC server peer1.freight.example.com url:grpcs://peer1.freight.example.com:10051 timeout:3000
2021-03-06T17:58:18.412Z - info: [NetworkConfig]: buildPeer - Unable to connect to the endorser peer1.freight.example.com due to Error: Failed to connect before the deadline on Endorser- name: peer1.freight.example.com, url:grpcs://peer1.freight.example.com:10051, connected:false, connectAttempted:true
2021-03-06T17:58:21.423Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Discoverer- name: peer0.freight.example.com, url:grpcs://peer0.freight.example.com:9051, connected:false, connectAttempted:true
2021-03-06T17:58:21.423Z - error: [ServiceEndpoint]: waitForReady - Failed to connect to remote gRPC server peer0.freight.example.com url:grpcs://peer0.freight.example.com:9051 timeout:3000
2021-03-06T17:58:21.423Z - error: [ServiceEndpoint]: ServiceEndpoint grpcs://peer0.freight.example.com:9051 reset connection failed :: Error: Failed to connect before the deadline on Discoverer- name: peer0.freight.example.com, url:grpcs://peer0.freight.example.com:9051, connected:false, connectAttempted:true
2021-03-06T17:58:24.425Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Discoverer- name: peer1.freight.example.com, url:grpcs://peer1.freight.example.com:10051, connected:false, connectAttempted:true
2021-03-06T17:58:24.425Z - error: [ServiceEndpoint]: waitForReady - Failed to connect to remote gRPC server peer1.freight.example.com url:grpcs://peer1.freight.example.com:10051 timeout:3000
2021-03-06T17:58:24.425Z - error: [ServiceEndpoint]: ServiceEndpoint grpcs://peer1.freight.example.com:10051 reset connection failed :: Error: Failed to connect before the deadline on Discoverer- name: peer1.freight.example.com, url:grpcs://peer1.freight.example.com:10051, connected:false, connectAttempted:true
2021-03-06T17:58:24.425Z - error: [DiscoveryService]: send[supplychain-channel] - no discovery results
Failed to evaluate transaction: function () Discovery Service has failed to return results

The network topology is 5 orgs with 2 orgs with 1 peer without orderer and 3 orgs with 2 peers each with orderer. Orderers are the part of peerOrganization itself. And each organization has its own CA. What can be the cause of this error?

Updated Discovery:

await gateway.connect(ccp, {
  wallet,
  identity: username,
  discovery: {
    enabled: true,
    asLocalhost: true
  },
  clientTlsIdentity: 'freight'
})

const network = await gateway.getNetwork('supplychain-channel')
console.log(network)

console:

Wallet Path: /home/hlfabric/fabric/api/freight/wallet
2021-03-10T04:54:26.827Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Committer- name: orderer0.freight.example.com:9050, url:grpcs://localhost:9050, connected:false, connectAttempted:true
2021-03-10T04:54:26.828Z - error: [ServiceEndpoint]: waitForReady - Failed to connect to remote gRPC server orderer0.freight.example.com:9050 url:grpcs://localhost:9050 timeout:3000
2021-03-10T04:54:26.828Z - error: [DiscoveryService]: _buildOrderer[supplychain-channel] - Unable to connect to the discovered orderer orderer0.freight.example.com:9050 due to Error: Failed to connect before the deadline on Committer- name: orderer0.freight.example.com:9050, url:grpcs://localhost:9050, connected:false, connectAttempted:true
2021-03-10T04:54:29.830Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Committer- name: orderer1.carrier.example.com:11050, url:grpcs://localhost:11050, connected:false, connectAttempted:true .....
NetworkImpl {
      contracts: Map {},
      initialized: true,
      commitListeners: Map {},
      blockListeners: Map {},
      gateway:
       Gateway {
         networks: Map { 'supplychain-channel' => [Circular] },
         options:
          { queryHandlerOptions: [Object],
            eventHandlerOptions: [Object],
            discovery: [Object],
            wallet: [Wallet],
            identity: 'sai',
            clientTlsIdentity: 'freight' },
         client:
          Client {
            type: 'Client',
            name: 'gateway client',
            mspid: null,
            _tls_mutual: [Object],
            endorsers: [Map],
            committers: [Map],
            channels: [Map],
            centralizedOptions: null },
         identity:
          { credentials: [Object], mspId: 'FreightMSP', type: 'X.509' },
         identityContext:
          IdentityContext {
            type: 'IdentityContext',
            client: [Client],
            user: [User],
            name: 'sai',
            mspid: 'FreightMSP',
            transactionId: null,
            nonce: null } },
      channel:
       Channel {
         type: 'Channel',
         name: 'supplychain-channel',
         client:
          Client {
            type: 'Client',
            name: 'gateway client',
            mspid: null,
            _tls_mutual: [Object],
            endorsers: [Map],
            committers: [Map],
            channels: [Map],
            centralizedOptions: null },
         endorsers:
          Map {
            'peer0.freight.example.com:9051' => [Endorser],
            'peer1.freight.example.com:10051' => [Endorser] },
         committers: Map { 'orderer0.freight.example.com:9050' => [Committer] },
         msps:
          Map {
            'FreightMSP' => [Object],
            'ConsigneeMSP' => [Object],
            'ShipperMSP' => [Object],
            'CarrierMSP' => [Object],
            'CustomMSP' => [Object] } },
      eventServiceManager:
       EventServiceManager {
         eventServices: Map {},
         network: [Circular],
         channel:
          Channel {
            type: 'Channel',
            name: 'supplychain-channel',
            client: [Client],
            endorsers: [Map],
            committers: [Map],
            msps: [Map] },
         mspId: 'FreightMSP',
         identityContext:
          IdentityContext {
            type: 'IdentityContext',
            client: [Client],
            user: [User],
            name: 'sai',
            mspid: 'FreightMSP',
            transactionId: null,
            nonce: null } },
      realtimeFilteredBlockEventSource:
       BlockEventSource {
         listeners: Set {},
         started: false,
         eventServiceManager:
          EventServiceManager {
            eventServices: Map {},
            network: [Circular],
            channel: [Channel],
            mspId: 'FreightMSP',
            identityContext: [IdentityContext] },
         blockQueue:
          OrderedBlockQueue { queue: Map {}, nextBlockNumber: undefined },
         asyncNotifier:
          AsyncNotifier {
            running: false,
            readCallback: [Function: bound getNextBlock],
            notifyCallback: [AsyncFunction: bound notifyListeners] },
         blockType: 'filtered' },
      realtimeFullBlockEventSource:
       BlockEventSource {
         listeners: Set {},
         started: false,
         eventServiceManager:
          EventServiceManager {
            eventServices: Map {},
            network: [Circular],
            channel: [Channel],
            mspId: 'FreightMSP',
            identityContext: [IdentityContext] },
         blockQueue:
          OrderedBlockQueue { queue: Map {}, nextBlockNumber: undefined },
         asyncNotifier:
          AsyncNotifier {
            running: false,
            readCallback: [Function: bound getNextBlock],
            notifyCallback: [AsyncFunction: bound notifyListeners] },
         blockType: 'full' },
      realtimePrivateBlockEventSource:
       BlockEventSource {
         listeners: Set {},
         started: false,
         eventServiceManager:
          EventServiceManager {
            eventServices: Map {},
            network: [Circular],
            channel: [Channel],
            mspId: 'FreightMSP',
            identityContext: [IdentityContext] },
         blockQueue:
          OrderedBlockQueue { queue: Map {}, nextBlockNumber: undefined },
         asyncNotifier:
          AsyncNotifier {
            running: false,
            readCallback: [Function: bound getNextBlock],
            notifyCallback: [AsyncFunction: bound notifyListeners] },
         blockType: 'private' },
      discoveryService:
       DiscoveryService {
         type: 'DiscoveryService',
         name: 'supplychain-channel',
         channel:
          Channel {
            type: 'Channel',
            name: 'supplychain-channel',
            client: [Client],
            endorsers: [Map],
            committers: [Map],
            msps: [Map] },
         client:
          Client {
            type: 'Client',
            name: 'gateway client',
            mspid: null,
            _tls_mutual: [Object],
            endorsers: [Map],
            committers: [Map],
            channels: [Map],
            centralizedOptions: null },
         refreshAge: undefined,
         discoveryResults:
          { msps: [Object],
            orderers: [Object],
            peers_by_org: [Object],
            timestamp: 1615352105872 },
         asLocalhost: true,
         currentTarget:
          Discoverer {
            name: 'peer0.freight.example.com',
            mspid: 'FreightMSP',
            client: [Client],
            connected: true,
            connectAttempted: true,
            endpoint: [Endpoint],
            service: [ServiceClientImpl],
            serviceClass: [Function],
            type: 'Discoverer',
            options: [Object] },
         targets: [ [Discoverer], [Discoverer] ],
         _action: { init: false, request: [Request] },
         _payload:
          <Buffer 0a c4 07 0a 9f 07 0a 0a 46 72 65 69 67 68 74 4d 53 50 12 90 07 2d 2d 2d 2d 2d 42 45 47 49 4e 20 43 45 52 54 49 46 49 43 41 54 45 2d 2d 2d 2d 2d 0a 4d ... >,
         _signature:
          <Buffer 30 44 02 20 18 dc 09 b5 2c 95 5d 98 14 c8 ee a0 47 48 f9 95 ae 74 1d b0 a9 d3 0e 21 cc dd 06 40 18 bc f4 f9 02 20 21 49 01 65 0b 2d 67 23 35 51 a8 dc ... >,
         requestTimeout: undefined },
      queryHandler:
       SingleQueryHandler { currentPeerIndex: 0, peers: [ [Endorser], [Endorser] ] } }

Running client in docker environment within same network the fabric is running solves the above issue

Dockerfile

FROM node:10.24
WORKDIR /usr/src/app
COPY package*.json ./
RUN apt-get update || : && apt-get install python -y
RUN npm install
COPY . .
EXPOSE 9080
CMD [ "node", "server.js" ]

First RUN is important because node image doesn’t include python and fabric node sdk requires python for some modules like pkcs11, resolves errors gyp ERR! find Python checking if "python3" can be used occurred while installing dependencies.

docker-compose.yml

version: "2.1"

networks: 
  supplychain:
    external: 
      name: supplychain

services: 
  api:
    image: api:1.0
    build: 
      context: "client/"
    networks: 
      - supplychain
    ports: 
      - 9080:9080

setting asLocalhost: false and urls of peers to "url": "grpcs://peer0.freight.example.com:9051" the peers are discovered by the client.

await gateway.connect(ccp, {
      wallet,
      identity: username,
      discovery: {
        enabled: true,
        asLocalhost: false
      }
    })