javadockergoogle-cloud-platformgoogle-publisher-taggoogle-cloud-pubsub-emulator

Google PubSub Emulator cannot list Project Subscriptions


I am using the Google PubSub emulator (locally with Docker) and trying to list the available subscriptions on my project. I am not using any authentication (it's just local development), using Docker and simple Java app, but I am running into this error:

Exception in thread "main" com.google.api.gax.rpc.PermissionDeniedException: io.grpc.StatusRuntimeException: PERMISSION_DENIED: The request is missing a valid API key.

This is my docker-compose code for the emulator:

gcloud:
    image: gcr.io/google.com/cloudsdktool/google-cloud-cli
    container_name: my-gcloud
    ports:
      - "8085:8085"
    entrypoint: gcloud beta emulators pubsub start --project emulator-project --host-port 0.0.0.0:8085 --verbosity=debug

This is how I create the topic (rest API, including a successful response):

curl -X PUT -H "Content-Type: application/json" http://localhost:8085/v1/projects/emulator-project/topics/test-topic
// response is:                                                                               
{
  "name": "projects/emulator-project/topics/test-topic"
}

I can retrieve the topic via REST and it's fine (so I assume the topic is succesfullly created):

curl -X GET http://localhost:8085/v1/projects/emulator-project/topics                                                                                                                             
    
{
  "topics": [{
    "name": "projects/emulator-project/topics/test-topic"
  }]
}

I add subscriptions also with the REST API:

curl -X POST \
  'http://localhost:8085/v1/projects/emulator-project/subscriptions' \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "projects/emulator-project/subscriptions/test-subscription",
    "topic": "projects/emulator-project/topics/test-topic"
  }'

However, using my Java code for retrieving the subscriptions inside the project, if any, I bump into the error(s) above:

 public class PubSubPublisherExample {

    public static void main(String[] args) throws Exception {

        try (SubscriptionAdminClient subscriptionAdminClient = SubscriptionAdminClient
                .create(SubscriptionAdminSettings.newBuilder()
                        .build())) {

            for (Subscription subscription : subscriptionAdminClient.listSubscriptions("emulator-project")
                    .iterateAll()) {
                System.out.println("Subscription: " + subscription.getName());
            }
        }
    }
}

I have tried not specifying the settings, or using the NoCredentials provider and if I do, then it somehow defaults to my production project account (although I have no environment variable or configuration to point to it, as I am testing in a simple Java app):

`Exception in thread "main" com.google.api.gax.rpc.PermissionDeniedException: io.grpc.StatusRuntimeException: PERMISSION_DENIED: Caller does not have required permission to use project SOME_PROD_PROJECT`.

Moreover there are no extra logs in the emulator container either.

Is there anyway I can simply use the local emulator project for any topic/subscription related interaction or is this an emulator limitation?


Solution

  • After some (more) days of searching, I have identified two main problems with my previous code. Posting my minimal solution here, as this might help other people starting with Google PubSub (and the emulator).

    1. The SubscriptionAdminClient needed to be created differently (at least to work with the emulator):

      String hostport = "localhost:8085";
      ManagedChannel channel = ManagedChannelBuilder.forTarget(hostport).usePlaintext().build();
      
      TransportChannelProvider channelProvider = FixedTransportChannelProvider.create(GrpcTransportChannel.create(channel));
      CredentialsProvider credentialsProvider = NoCredentialsProvider.create();
      
      SubscriptionAdminClient subscriptionAdminClient
                   = SubscriptionAdminClient.create(SubscriptionAdminSettings.newBuilder()
                   .setTransportChannelProvider(channelProvider)
                   .setCredentialsProvider(credentialsProvider)
                   .build()
           );
      
    1. The listing of subscriptions did not work with a simple String as project name (at least for me). Using a ProjectName instance fixed this too:

      ProjectName projectName = ProjectName.of("emulator-project");
      for (Subscription subscription : subscriptionAdminClient.listSubscriptions(projectName).iterateAll()) {
          System.out.println("Existing subscriptions: " + subscription.getName());
      }
      

    With the code above, I no longer get API key (or credentials) related warnings/errors. This is enough to get my sample going.

    (Again I needed to use this just for a small example, not for something production ready)

    PS Using a similar way of creating the TopicAdminClient, I was able to created topics via code (not just through the REST API) and with this SubscriptionAdminClient I was able to create subscriptions in code too. In the end, I did not require the REST API, but it was useful for simple logging/debugging at first.