google-cloud-platformgoogle-cloud-functionsgoogle-cloud-pubsubgoogle-cloud-iot

subFolder is empty when using a Google IoT Core gateway and Pub/Sub


I have a device publishing through a gateway on the events topic (/devices/<dev_id>/events/motion) to PubSub. It's landing in PubSub correctly but subFolder is just an empty string.

On the gateway I'm publishing using the code below. f"mb.{device_id}" is the device_id (not the gateway ID and attribute could be anything - motion, temperature, etc

def report(self, device_id, attribute, value):
    topic = f"/devices/mb.{device_id}/events/{attribute}"
    timestamp = datetime.utcnow().timestamp()
    client.publish(topic, json.dumps({"v": value, "ts": timestamp}))

And this is the cloud function listening on the PubSub queue.

def iot_to_bigtable(event, context):
    payload = json.loads(base64.b64decode(event["data"]).decode("utf-8"))
    timestamp = payload.get("ts")
    value = payload.get("v")
    if not timestamp or value is None:
        raise BadDataException()
    attributes = event.get("attributes", {})
    device_id = attributes.get("deviceId")
    registry_id = attributes.get("deviceRegistryId")
    attribute = attributes.get("subFolder")
    if not device_id or not registry_id or not attribute:
        raise BadDataException()

A sample of the event in Pub/Sub:

{
    @type: 'type.googleapis.com/google.pubsub.v1.PubsubMessage', 
    attributes: {
        deviceId: 'mb.26727bab-0f37-4453-82a4-75d93cb3f374', 
        deviceNumId: '2859313639674234', 
        deviceRegistryId: 'mb-staging', 
        deviceRegistryLocation: 'europe-west1', 
        gatewayId: 'mb.42e29cd5-08ad-40cf-9c1e-a1974144d39a', 
        projectId: 'mb-staging', 
        subFolder: ''
    }, 
    data: 'eyJ2IjogImxvdyIsICJ0cyI6IDE1OTA3NjgzNjcuMTMyNDQ4fQ=='
}

Why is subFolder empty? Based on the docs I was expecting it to be the attribute (i.e. motion or temperature)


Solution

  • This issue has nothing to do with Cloud IoT Core. It is instead caused by how Pub/Sub handles failed messages. It was retrying messages from ~12 hours ago that had failed (and didn't have an attribute).

    You fix this by purging the Subscription in Pub/Sub.