I run the code:https://github.com/prometheus-community/json_exporter by changing config to convert json data to prometheus metrics data type
The error is:
level=error ts=2021-07-05T06:19:52.714Z caller=main.go:58 msg="Error loading config" err="yaml: unmarshal errors:\n line 7: cannot unmarshal !!str `id:\"{.id}\"` into map[string]string"
Here is my yaml config:
metrics:
- name: running_applications
type: object
help: Information on running applications
path: '{.apps.app[?(@.state=="RUNNING")]}'
labels:
id:'{.id}'
values:
memory: '{.allocatedMB}'
vcores: '{.allocatedVCores}'
Here is part of json:
{"apps":
{"app":
[{"id":"application_1622098898935_1480","user":"hive","name":"HIVE-b5770b0d-fdd0-4ee4-86f8-1684df70dae9","queue":"default","state":"FINISHED","finalStatus":"SUCCEEDED","progress":100.0,"trackingUI":"History","trackingUrl":"http://tst02.bj2.sdbmcc.com:30999/proxy/application_1622098898935_1480/","diagnostics":"Attempt recovered after RM restartSession timed out, lastDAGCompletionTime=1624665783612 ms, sessionTimeoutInterval=600000 ms\nSession stats:submittedDAGs=1, successfulDAGs=1, failedDAGs=0, killedDAGs=0\n","clusterId":1625045401338,"applicationType":"TEZ","applicationTags":"hive_20210626080103_7098ab3b-ea51-4780-ba5d-aa823335b102,userid=user-app_ebupt","priority":0,"startedTime":1624665666232,"finishedTime":1624666419590,"elapsedTime":753358,"amContainerLogs":"http://tst02.bj2.sdbmcc.com:8042/node/containerlogs/container_e82_1622098898935_1480_01_000001/hive","amHostHttpAddress":"tst02.bj2.sdbmcc.com:8042","masterNodeId":"tst02.bj2.sdbmcc.com:45454","allocatedMB":-1,"allocatedVCores":-1,"reservedMB":-1,"reservedVCores":-1,"runningContainers":-1,"memorySeconds":3752447,"vcoreSeconds":915,"queueUsagePercentage":0.0,"clusterUsagePercentage":0.0,"resourceSecondsMap":{"entry":{"key":"memory-mb","value":"3752447"},"entry":{"key":"vcores","value":"915"}},"preemptedResourceMB":0,"preemptedResourceVCores":0,"numNonAMContainerPreempted":0,"numAMContainerPreempted":0,"preemptedMemorySeconds":3752447,"preemptedVcoreSeconds":915,"preemptedResourceSecondsMap":{"entry":{"key":"memory-mb","value":"3752447"},"entry":{"key":"vcores","value":"915"}},"logAggregationStatus":"NOT_START","unmanagedApplication":false,"amNodeLabelExpression":"","timeouts":{"timeout":[{"type":"LIFETIME","expiryTime":"UNLIMITED","remainingTimeInSeconds":-1}]}},
{"id":"application_1622098898935_1479","user":"hive","name":"HIVE-db17be65-b129-497b-a89c-28b001acf3fc","queue":"default","state":"FINISHED","finalStatus":"SUCCEEDED","progress":100.0,"trackingUI":"History","trackingUrl":"http://tst02.bj2.sdbmcc.com:30999/proxy/application_1622098898935_1479/","diagnostics":"Attempt recovered after RM restartSession timed out, lastDAGCompletionTime=1624663983139 ms, sessionTimeoutInterval=600000 ms\nSession stats:submittedDAGs=1, successfulDAGs=1, failedDAGs=0, killedDAGs=0\n","clusterId":1625045401338,"applicationType":"TEZ","applicationTags":"hive_20210626073104_b7560acd-42db-46b8-bbd2-72e4f0599624,userid=user-app_ebupt","priority":0,"startedTime":1624663872431,"finishedTime":1624664618529,"elapsedTime":746098,"amContainerLogs":"http://tst06.bj2.sdbmcc.com:8042/node/containerlogs/container_e82_1622098898935_1479_01_000001/hive","amHostHttpAddress":"tst06.bj2.sdbmcc.com:8042","masterNodeId":"tst06.bj2.sdbmcc.com:45454","allocatedMB":-1,"allocatedVCores":-1,"reservedMB":-1,"reservedVCores":-1,"runningContainers":-1,"memorySeconds":3802102,"vcoreSeconds":927,"queueUsagePercentage":0.0,"clusterUsagePercentage":0.0,"resourceSecondsMap":{"entry":{"key":"memory-mb","value":"3802102"},"entry":{"key":"vcores","value":"927"}},"preemptedResourceMB":0,"preemptedResourceVCores":0,"numNonAMContainerPreempted":0,"numAMContainerPreempted":0,"preemptedMemorySeconds":3802102,"preemptedVcoreSeconds":927,"preemptedResourceSecondsMap":{"entry":{"key":"memory-mb","value":"3802102"},"entry":{"key":"vcores","value":"927"}},"logAggregationStatus":"NOT_START","unmanagedApplication":false,"amNodeLabelExpression":"","timeouts":{"timeout":[{"type":"LIFETIME","expiryTime":"UNLIMITED","remainingTimeInSeconds":-1}]}},
{"id":"application_1622098898935_1478","user":"hive","name":"HIVE-2805816c-9723-4d32-90f7-146d7bf61946","queue":"default","state":"FINISHED","finalStatus":"SUCCEEDED","progress":100.0,"trackingUI":"History","trackingUrl":"http://tst02.bj2.sdbmcc.com:30999/proxy/application_1622098898935_1478/","diagnostics":"Attempt recovered after RM restartSession timed out, lastDAGCompletionTime=1624662198199 ms, sessionTimeoutInterval=600000 ms\nSession stats:submittedDAGs=1, successfulDAGs=1, failedDAGs=0, killedDAGs=0\n","clusterId":1625045401338,"applicationType":"TEZ","applicationTags":"hive_20210626070103_77d85e4a-ead5-43d1-b36f-8c766719c551,userid=user-app_ebupt","priority":0,"startedTime":1624662071009,"finishedTime":1624662813506,"elapsedTime":742497,"amContainerLogs":"http://tst02.bj2.sdbmcc.com:8042/node/containerlogs/container_e82_1622098898935_1478_01_000001/hive","amHostHttpAddress":"tst02.bj2.sdbmcc.com:8042","masterNodeId":"tst02.bj2.sdbmcc.com:45454","allocatedMB":-1,"allocatedVCores":-1,"reservedMB":-1,"reservedVCores":-1,"runningContainers":-1,"memorySeconds":3947826,"vcoreSeconds":962,"queueUsagePercentage":0.0,"clusterUsagePercentage":0.0,"resourceSecondsMap":{"entry":{"key":"memory-mb","value":"3947826"},"entry":{"key":"vcores","value":"962"}},"preemptedResourceMB":0,"preemptedResourceVCores":0,"numNonAMContainerPreempted":0,"numAMContainerPreempted":0,"preemptedMemorySeconds":3947826,"preemptedVcoreSeconds":962,"preemptedResourceSecondsMap":{"entry":{"key":"memory-mb","value":"3947826"},"entry":{"key":"vcores","value":"962"}},"logAggregationStatus":"NOT_START","unmanagedApplication":false,"amNodeLabelExpression":"","timeouts":{"timeout":[{"type":"LIFETIME","expiryTime":"UNLIMITED","remainingTimeInSeconds":-1}]}},
{"id":"application_1622098898935_1477","user":"hive","name":"HIVE-df9d6d21-5011-41f7-8482-dc3f214b6d38","queue":"default","state":"FINISHED","finalStatus":"SUCCEEDED","progress":100.0,"trackingUI":"History","trackingUrl":"http://tst02.bj2.sdbmcc.com:30999/proxy/application_1622098898935_1477/","diagnostics":"Attempt recovered after RM restartSession timed out, lastDAGCompletionTime=1624660381598 ms, sessionTimeoutInterval=600000 ms\nSession stats:submittedDAGs=1, successfulDAGs=1, failedDAGs=0, killedDAGs=0\n","clusterId":1625045401338,"applicationType":"TEZ","applicationTags":"hive_20210626063103_be7a2ca1-35a3-4435-8db5-60e72b97c076,userid=user-app_ebupt","priority":0,"startedTime":1624660266652,"finishedTime":1624661012324,"elapsedTime":745672,"amContainerLogs":"http://tst06.bj2.sdbmcc.com:8042/node/containerlogs/container_e82_1622098898935_1477_01_000001/hive","amHostHttpAddress":"tst06.bj2.sdbmcc.com:8042","masterNodeId":"tst06.bj2.sdbmcc.com:45454","allocatedMB":-1,"allocatedVCores":-1,"reservedMB":-1,"reservedVCores":-1,"runningContainers":-1,"memorySeconds":3813440,"vcoreSeconds":930,"queueUsagePercentage":0.0,"clusterUsagePercentage":0.0,"resourceSecondsMap":{"entry":{"key":"memory-mb","value":"3813440"},"entry":{"key":"vcores","value":"930"}},"preemptedResourceMB":0,"preemptedResourceVCores":0,"numNonAMContainerPreempted":0,"numAMContainerPreempted":0,"preemptedMemorySeconds":3813440,"preemptedVcoreSeconds":930,"preemptedResourceSecondsMap":{"entry":{"key":"memory-mb","value":"3813440"},"entry":{"key":"vcores","value":"930"}},"logAggregationStatus":"NOT_START","unmanagedApplication":false,"amNodeLabelExpression":"","timeouts":{"timeout":[{"type":"LIFETIME","expiryTime":"UNLIMITED","remainingTimeInSeconds":-1}]}},
{"id":"application_1622098898935_1476","user":"hive","name":"HIVE-114bdd08-b399-4c2f-8ec9-6320985f612a","queue":"default","state":"FINISHED","finalStatus":"SUCCEEDED","progress":100.0,"trackingUI":"History","trackingUrl":"http://tst02.bj2.sdbmcc.com:30999/proxy/application_1622098898935_1476/","diagnostics":"Attempt recovered after RM restartSession timed out, lastDAGCompletionTime=1624658578652 ms, sessionTimeoutInterval=600000 ms\nSession stats:submittedDAGs=1, successfulDAGs=1, failedDAGs=0, killedDAGs=0\n","clusterId":1625045401338,"applicationType":"TEZ","applicationTags":"hive_20210626060103_1d7f2f72-62de-4d27-be50-0625bbc10409,userid=user-app_ebupt","priority":0,"startedTime":1624658466988,"finishedTime":1624659212326,"elapsedTime":745338,"amContainerLogs":"http://tst02.bj2.sdbmcc.com:8042/node/containerlogs/container_e82_1622098898935_1476_01_000001/hive","amHostHttpAddress":"tst02.bj2.sdbmcc.com:8042","masterNodeId":"tst02.bj2.sdbmcc.com:45454","allocatedMB":-1,"allocatedVCores":-1,"reservedMB":-1,"reservedVCores":-1,"runningContainers":-1,"memorySeconds":3818593,"vcoreSeconds":931,"queueUsagePercentage":0.0,"clusterUsagePercentage":0.0,"resourceSecondsMap":{"entry":{"key":"memory-mb","value":"3818593"},"entry":{"key":"vcores","value":"931"}},"preemptedResourceMB":0,"preemptedResourceVCores":0,"numNonAMContainerPreempted":0,"numAMContainerPreempted":0,"preemptedMemorySeconds":3818593,"preemptedVcoreSeconds":931,"preemptedResourceSecondsMap":{"entry":{"key":"memory-mb","value":"3818593"},"entry":{"key":"vcores","value":"931"}},"logAggregationStatus":"NOT_START","unmanagedApplication":false,"amNodeLabelExpression":"","timeouts":{"timeout":[{"type":"LIFETIME","expiryTime":"UNLIMITED","remainingTimeInSeconds":-1}]}}
]
}
}
Based on the suggestion of marco.m ,modify the yaml configuration file as follows:
metrics:
- name: running_applications
type: object
help: Information on running applications
path: '{.apps.app[?(@.state=="RUNNING")]}'
labels:
id: '{.id}'
values:
memory: '{.allocatedMB}'
vcores: '{.allocatedVCores}'