Google Vision API with Multi-regional support: gives StatusRuntimeException: UNAUTHENTICATED when using with custom endpoint

Getting the StatusRuntimeException: UNAUTHENTICATED when using with custom endpoint as per the following guide: Google Vision API - Multi-regional support

But all works fine when using default endpoint. See the code sample below:

Credentials credentials = ServiceAccountCredentials.fromStream(new FileInputStream(credPath));

// Works fine
ImageAnnotatorSettings settings = ImageAnnotatorSettings.newBuilder().setCredentialsProvider(FixedCredentialsProvider.create(credentials)).build();

// Works fine as well
ImageAnnotatorSettings settings = ImageAnnotatorSettings.newBuilder().setCredentialsProvider(FixedCredentialsProvider.create(credentials)).setEndpoint("vision.googleapis.com:443").build();

// Doesn't work - throws exception:
ImageAnnotatorSettings settings = ImageAnnotatorSettings.newBuilder().setCredentialsProvider(FixedCredentialsProvider.create(credentials)).setEndpoint("us-vision.googleapis.com:443").build();

visionClient = ImageAnnotatorClient.create(settings);

Exception is the following:

com.google.api.gax.rpc.UnauthenticatedException: io.grpc.StatusRuntimeException: UNAUTHENTICATED: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.
    at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:73)
    at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:72)
    at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:60)
    at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)
    at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68)
    at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1074)
    at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
    at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213)
    at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983)
    at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771)
    at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:563)
    at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:533)
    at io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:463)
    at io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:427)
    at io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:460)
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:557)
    at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:69)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:738)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:717)
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
    at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
    Suppressed: com.google.api.gax.rpc.AsyncTaskException: Asynchronous task failed
        at com.google.api.gax.rpc.ApiExceptions.callAndTranslateApiException(ApiExceptions.java:57)
        at com.google.api.gax.rpc.UnaryCallable.call(UnaryCallable.java:112)
        at com.google.cloud.vision.v1.ImageAnnotatorClient.batchAnnotateImages(ImageAnnotatorClient.java:203)
        at com.google.cloud.vision.v1.ImageAnnotatorClient.batchAnnotateImages(ImageAnnotatorClient.java:179)
        ...
        at com.sun.ejb.containers.EjbAsyncTask.call(EjbAsyncTask.java:101)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        ... 3 more
Caused by: io.grpc.StatusRuntimeException: UNAUTHENTICATED: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.
    at io.grpc.Status.asRuntimeException(Status.java:535)
    ... 13 more

Any thoughts how to fix that? Thanks!

Solution

Just in case it might help someone, I had this issue: Google DocumentAI Java example fails with io.grpc.StatusRuntimeException: INVALID_ARGUMENT: Request contains an invalid argument THEN the same issue described here that I solved this way:

I tried with the original way of authentication provided in the documentation:

export GOOGLE_APPLICATION_CREDENTIALS=/path/to/my/key.json

It worked with this authentication method, but as soon as I added the credentials manually it stopped working:

.setCredentialsProvider(FixedCredentialsProvider.create(credentials))

I decided to log both credentials using settingsBuilder.getCredentialsProvider, it appeared that the automatic credential set a custom scope automatically "https://www.googleapis.com/auth/cloud-platform" that is NOT indicated in the generated key.kson file (scope array was empty). And it is NOT indicated that we must add it when authenticating through an InputStream in the google documentation https://github.com/googleapis/google-cloud-java#authentication as of today:

.setCredentials(ServiceAccountCredentials.fromStream(new FileInputStream("/path/to/my/key.json")))

So I wrote this code instead:

Credentials credentials = 
            ServiceAccountCredentials
              .fromStream(credentialStream)
              .createScoped("https://www.googleapis.com/auth/cloud-platform")

And it worked ! It could also work if you edit the key.json file to add the scope in the array, I haven't tested it.

Both issue came from a lack of documentation in the Java libraries, so If someone from Google read this answer, please complete the documentation.