azure-iot-hubazure-stream-analyticsiotplugandplay

How to get dt-subject system property from Azure IoT Hub into Azure Stream Analytics


Background: I am using Azure Stream Analytics to process telemetry events from IoT Hub. The telemetry messages themselves include a dt-subject system property, because the telemetry corresponds to a component in an IoT Plug and Play model.

Problem: The dt-subject system property is not received by Azure Stream Analytics.

Question: Can this be solved simply through configuration?

Here's what one telemetry message looks like when viewed from IoT Explorer:

{
  "body": {
    "temperature": 3645.08
  },
  "enqueuedTime": "Wed Mar 22 2023 09:43:03 GMT-0700 (Pacific Daylight Time)",
  "sequenceNumber": 11890,
  "systemProperties": {
    "iothub-connection-device-id": "tc-win-02",
    "iothub-connection-auth-method": "{\"scope\":\"device\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}",
    "iothub-connection-auth-generation-id": "638103031903876646",
    "iothub-enqueuedtime": 1679503383862,
    "iothub-message-source": "Telemetry",
    "dt-subject": "thermostat2",
    "dt-dataschema": "dtmi:com:example:TemperatureController;2",
    "contentType": "application/json",
    "contentEncoding": "utf-8"
  }
}

Here's what it looks like coming IN to stream analytics. This is taken from the Input Preview window in the Job Topology / Query pane of Steam Analytics Job portal. For testing, I saved these out to Table Storage using Azure Stream Analytics, which looks just like this raw dump would suggest.

Notice there is no dt-subject property to be found.

  {
    "temperature": 3647,
    "EventProcessedUtcTime": "2023-03-22T17:12:36.9899389Z",
    "PartitionId": 1,
    "EventEnqueuedUtcTime": "2023-03-22T16:44:55.5450000Z",
    "IoTHub": {
      "MessageId": null,
      "CorrelationId": null,
      "ConnectionDeviceId": "tc-win-02",
      "ConnectionDeviceGenerationId": "638103031903876646",
      "EnqueuedTime": "2023-03-22T16:44:55.4270000Z"
    }
  },

For one more test, I routed the telemetry messages directly to a blob storage container from IoT Hub. This is what they look like there. Does include dt-subject there.

{
    "EnqueuedTimeUtc": "2023-03-22T16:53:03.0520000Z",
    "Properties": {},
    "SystemProperties": {
        "connectionDeviceId": "tc-win-02",
        "connectionAuthMethod": "{\"scope\":\"device\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}",
        "connectionDeviceGenerationId": "638103031903876646",
        "contentType": "application/json",
        "contentEncoding": "utf-8",
        "dt-subject": "thermostat2",
        "dt-dataschema": "dtmi:com:example:TemperatureController;2",
        "enqueuedTime": "2023-03-22T16:53:03.0520000Z"
    },
    "Body": {
        "temperature": 3655.08
    }
}

Also, I did try GetMetadataPropertyValue. In the query. I added these lines to the SELECT clause. As the documentation suggests, these columns showed up as "null" in the Test results pane. However, the columns are never added to the Table Storage output.

    GetMetadataPropertyValue([input-sp4mvwjwloirs], '[EventHub].[IoTMessageSource]') AS source,
    GetMetadataPropertyValue([input-sp4mvwjwloirs], '[EventHub].[IoTInterfaceName]') AS interface,


Solution

  • The following example shows how to obtain a device modelId property:

    SELECT GetMetadataPropertyValue(inpsim, '[IoTHub].DataSchema') AS ModelId INTO outBlob FROM inpsim