google-cloud-platformgrpcgoogle-cloud-mlgrpc-pythongoogle-cloud-vertex-ai

_InactiveRpcError while querying Vertex AI Matching Engine Index


I am following the example notebook as per GCP docs to test Vertex Matching Engine. I have deployed an index but while trying to query the index I am getting _InactiveRpcError. The VPC network is in us-west2 with private service access enabled and the Index is deployed in us-central1. My VPC network contains the pre-populated firewall rules.

Index

createTime: '2021-11-23T15:25:53.928606Z'
deployedIndexes:
- deployedIndexId: brute_force_glove_deployed_v3
  indexEndpoint: projects/XXXXXXXXXXXX/locations/us-central1/indexEndpoints/XXXXXXXXXXXX
description: testing python script for creating index
displayName: glove_100_brute_force_20211123152551
etag: AMEw9yOVPWBOTpbAvJLllqxWMi2YurEV_sad2n13QvbIlqjOdMyiq_j20gG1ldhdZNTL
metadata:
  config:
    algorithmConfig:
      bruteForceConfig: {}
    dimensions: 100
    distanceMeasureType: DOT_PRODUCT_DISTANCE
metadataSchemaUri: gs://google-cloud-aiplatform/schema/matchingengine/metadata/nearest_neighbor_search_1.0.0.yaml
name: projects/XXXXXXXXXXXX/locations/us-central1/indexes/XXXXXXXXXXXX
updateTime: '2021-11-23T16:04:17.993730Z'

Index-Endpoint

createTime: '2021-11-24T10:59:51.975949Z'
deployedIndexes:
- automaticResources:
    maxReplicaCount: 1
    minReplicaCount: 1
  createTime: '2021-11-30T15:16:12.323028Z'
  deploymentGroup: default
  displayName: brute_force_glove_deployed_v3
  enableAccessLogging: true
  id: brute_force_glove_deployed_v3
  index: projects/XXXXXXXXXXXX/locations/us-central1/indexes/XXXXXXXXXXXX
  indexSyncTime: '2021-11-30T16:37:35.597200Z'
  privateEndpoints:
    matchGrpcAddress: 10.242.4.5
displayName: index_endpoint_for_demo
etag: AMEw9yO6cuDfgpBhGVw7-NKnlS1vdFI5nnOtqVgW1ddMP-CMXM7NfGWVpqRpMRPsNCwc
name: projects/XXXXXXXXXXXX/locations/us-central1/indexEndpoints/XXXXXXXXXXXX
network: projects/XXXXXXXXXXXX/global/networks/XXXXXXXXXXXX
updateTime: '2021-11-24T10:59:53.271100Z'

Code


import grpc

# import the generated classes
import match_service_pb2
import match_service_pb2_grpc

DEPLOYED_INDEX_SERVER_IP = '10.242.0.5'
DEPLOYED_INDEX_ID = 'brute_force_glove_deployed_v3'

query = [-0.11333, 0.48402, 0.090771, -0.22439, 0.034206, -0.55831, 0.041849, -0.53573, 0.18809, -0.58722, 0.015313, -0.014555, 0.80842, -0.038519, 0.75348, 0.70502, -0.17863, 0.3222, 0.67575, 0.67198, 0.26044, 0.4187, -0.34122, 0.2286, -0.53529, 1.2582, -0.091543, 0.19716, -0.037454, -0.3336, 0.31399, 0.36488, 0.71263, 0.1307, -0.24654, -0.52445, -0.036091, 0.55068, 0.10017, 0.48095, 0.71104, -0.053462, 0.22325, 0.30917, -0.39926, 0.036634, -0.35431, -0.42795, 0.46444, 0.25586, 0.68257, -0.20821, 0.38433, 0.055773, -0.2539, -0.20804, 0.52522, -0.11399, -0.3253, -0.44104, 0.17528, 0.62255, 0.50237, -0.7607, -0.071786, 0.0080131, -0.13286, 0.50097, 0.18824, -0.54722, -0.42664, 0.4292, 0.14877, -0.0072514, -0.16484, -0.059798, 0.9895, -0.61738, 0.054169, 0.48424, -0.35084, -0.27053, 0.37829, 0.11503, -0.39613, 0.24266, 0.39147, -0.075256, 0.65093, -0.20822, -0.17456, 0.53571, -0.16537, 0.13582, -0.56016, 0.016964, 0.1277, 0.94071, -0.22608, -0.021106]

channel = grpc.insecure_channel("{}:10000".format(DEPLOYED_INDEX_SERVER_IP))
stub = match_service_pb2_grpc.MatchServiceStub(channel)

request = match_service_pb2.MatchRequest()
request.deployed_index_id = DEPLOYED_INDEX_ID
for val in query:
    request.float_val.append(val)

response = stub.Match(request)
response

Error

_InactiveRpcError                         Traceback (most recent call last)
/tmp/ipykernel_3451/467153318.py in <module>
    108     request.float_val.append(val)
    109 
--> 110 response = stub.Match(request)
    111 response

/opt/conda/lib/python3.7/site-packages/grpc/_channel.py in __call__(self, request, timeout, metadata, credentials, wait_for_ready, compression)
    944         state, call, = self._blocking(request, timeout, metadata, credentials,
    945                                       wait_for_ready, compression)
--> 946         return _end_unary_response_blocking(state, call, False, None)
    947 
    948     def with_call(self,

/opt/conda/lib/python3.7/site-packages/grpc/_channel.py in _end_unary_response_blocking(state, call, with_call, deadline)
    847             return state.response
    848     else:
--> 849         raise _InactiveRpcError(state)
    850 
    851 

_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    status = StatusCode.UNAVAILABLE
    details = "failed to connect to all addresses"
    debug_error_string = "{"created":"@1638277076.941429628","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3093,"referenced_errors":[{"created":"@1638277076.941428202","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":163,"grpc_status":14}]}"
>

Solution

  • Currently, Matching Engine only supports Query from the same region. Can you try running the code from VM in us-central1.