GCP Pub/Sub order key property not working

I am using a python client for the GCP Pub/Sub and FastAPI. I have a push subscription to an endpoint and am sending messages with the order key property. However, about one out of 4 times, I receive the messages out of order on the receiving endpoint.

I have a Pub/Sub subscription with the order key property enabled: screenshot of subscription details

I am using the python client for GCP Pub/Sub to publish 2 messages:

from google.cloud import pubsub
import json
import typing

pubsub_client = pubsub.PublisherClient()


def fire_event_to_pubsub(
    event_topic_name: str,
    event_payload: dict[str, typing.Any],
    order_key: str,
) -> None:
    # Publish the event
    topic_path = pubsub_client.topic_path(
        project="project_name", topic=event_topic_name
    )
    message_json = json.dumps(event_payload)
    # Encode the JSON string to bytes
    message_bytes = message_json.encode("utf-8")
    future = pubsub_client.publish(topic_path, data=message_bytes, order_key=order_key)
    print(future.result())
    print(
        f"Published message to topic {topic_path}: {message_json} with order key {order_key}"
    )


form_payload1 = {
    "application_id": "50030469",
    "form_path": "gs://fa-form-received-14534/application1/50030469.PDF",
    "form_hash": "1111",
}

fire_event_to_pubsub(
    "form_received_ordered",
    form_payload1,
    "50030469" + "-" + "form_name",
)

form_payload2 = {
    "application_id": "50030469",
    "form_path": "gs://fa-form-received-14534/application2/50030469.PDF",
    "form_hash": "2222",
}
fire_event_to_pubsub(
    "form_received_ordered",
    form_payload2,
    "50030469" + "-" + "form_name",
)

There is a FastAPI endpoint that logs the messages. However-- they appear to be out of order in the logs. Here you can see the order key property being used-- and the publish time is slightly earlier on the 2nd message-- so I would expect it to be logged first: screenshot of logs

Has anybody encountered this problem before? Is there something I am missing?

The property for publishing with an ordering key is ordering_key, not order_key. By using order_key instead in publish, the publisher is treating the field as an attribute instead of putting it in the ordering_key property, which means no delivery order is guaranteed. You should also enable ordering in the publisher client itself in the PublisherOptions. This wouldn't likely affect your code as written because you wait for the future result of one publish before publishing the next, but if you ever waited on the result asynchronously, not setting enable_message_ordering could result in out-of-order publishing.

See the sample code for publishing in Python with an ordering key, most importantly:

publisher_options = pubsub_v1.types.PublisherOptions(enable_message_ordering=True)
publisher = pubsub_v1.PublisherClient(publisher_options=publisher_options)
...
future = publisher.publish(topic_path, data=data, ordering_key=ordering_key)