So, the scenario is simple: I want to configure google's pubsub emulator to have a dead letter queue topic, and if a message results in an exception in the handler that message should be forwarded to the dead letter queue topic. I then want to be able to subscribe to that topic and pull dead letters from it. I'm using spring-cloud-gcp-starter-pubsub in my producers and consumers, then using the following shell commands to launch and configure the emulator:
gcloud beta emulators pubsub start --project=local-test --host-port=0.0.0.0:8080
then after that stands up fully,
curl -X PUT -v http://localhost:8080/v1/projects/local-test/topics/fam-task
curl -X PUT -v http://localhost:8080/v1/projects/local-test/topics/dead-letter
curl -X PUT 'http://localhost:8080/v1/projects/local-test/subscriptions/subscription-task-reactor-fam-task' \
--header 'Content-Type: application/json' --data '{"topic": "projects/local-test/topics/fam-task", "enableMessageOrdering": true, "enableExactlyOnceDelivery": true, "deadLetterPolicy": {"deadLetterTopic": "projects/local-test/topics/dead-letter", "maxDeliveryAttempts": 5}}'
echo "Create dead letter subscriptions"
curl -X PUT 'http://localhost:8080/v1/projects/local-test/subscriptions/subscription-dead-letter' \
--header 'Content-Type: application/json' --data '{"topic": "projects/local-test/topics/dead-letter", "enableMessageOrdering": true, "enableExactlyOnceDelivery": true}'
I'm able to verify producers and consumers are wired together correctly and sending/receiving messages as expected. For proof of concept purposes I'm having the receiving method just throw a new RuntimeException() when a message is received. When that happens, my app logs show
PubSubInboundChannelAdapter - Sending Spring message [4] failed; message nacked automatically.
5 times, which matches the expected behavior as defined when I create subscription-task-reactor-fam-task, but polling the dead letter subscription by hitting
POST http://localhost:8080/v1/projects/local-test/subscriptions/subscription-dead-letter:pull
results in a timeout with no messages returned. The subscription and topic are there, the dead letter topic is there and seems to be designated in the creation of the subscription. The subscription to the dead letter topic is there, valid and pollable, it's just that even when messages get nacked they don't show up when I poll the dead letter topic. I see other questions with answers about making sure the service account has the subscriber and publisher roles available to it, but I'm not sure how (if at all) that translates to the emulator. Any idea what I'm doing wrong?
When I changed enableMessageOrdering to false for subscription-task-reactor-fam-task messages began arriving correctly in the DLT and messages arriving after the DLT'd message were delivered normally.