google-cloud-platformgoogle-cloud-pubsubunordered

Cloud Pub/Sub - Single message delaying others (HoL blocking?)


Today I experienced something I found rather interesting. I had a batch of unacknowledged messages that were all published within the same second, and for an expected reason, one of these messages were being unacknowledged. However, the remaining messages kept being attempted delivered and were being processed and acknowledged successfully.

Why does this happen? Is this expected behavior? The messages did not have an ordering key, nor was message ordering enabled on the given subscription.

Also, I even attempted to ACK these messages manually in Google Cloud, but it did not seem to do anything. When I pulled after ACKing, the same messages showed up.


Solution

  • You are probably running into the case described in the note in the "dealing with duplicates" section of the documentation. If messages are batched together, all messages in the batch must be acknowledged or the entire batch of messages may be redelivered. This means that if 100 messages were batched together in a single publish request and 99 of them are acked, but 1 is not acked, all 100 may be redelivered. There are some efforts to avoid this duplicate delivery as much as possible in the service, but it is not guaranteed.