i want to create a pipeline to read XML data from postman http url and consume it through benthos input configuration and publish this message to a kafka topic using benthos processor . Following is the configuration , i was trying but doesn't seem working
input:
http_server:
address: ""
path: /
ws_path: /ws
allowed_verbs:
- POST
timeout: 5s
rate_limit: ""
pipeline:
processors:
- xml: {}
output:
kafka:
addresses:
- abc-central-1-kafka-2-client.abc.svc.prod1-us-central1.gke.kaas-prod-us.gcp.extscloud.com:16552
- abc-central-1-kafka-1-client.abc.svc.prod1-us-central1.gke.kaas-prod-us.gcp.extscloud.com:16552
- abc-central-1-kafka-0-client.abc.svc.prod1-us-central1.gke.kaas-prod-us.gcp.extscloud.com:16552
topic: abc_abc-central-1.abc-tyur-service-in.v1
client_id: abc-tyur-service-dev-1
tls:
enabled: true
root_cas_file: /Users/ca.crt
client_certs:
- cert_file: /Users/cert.pem
key_file: /Users/key.pem
logger:
level: ALL
When we don't want to modify our input messages and want to publish unmodified messages to the output , processor can be removed and only input and output configuration are sufficient. Removing processor solved my problem . Following the working configuration
input:
http_server:
address: ""
path: /
ws_path: /ws
allowed_verbs:
- POST
timeout: 5s
rate_limit: ""
output:
kafka:
addresses:
- abc-central-1-kafka-2-client.abc.svc.prod1-us-central1.gke.kaas-prod-us.gcp.extscloud.com:16552
- abc-central-1-kafka-1-client.abc.svc.prod1-us-central1.gke.kaas-prod-us.gcp.extscloud.com:16552
- abc-central-1-kafka-0-client.abc.svc.prod1-us-central1.gke.kaas-prod-us.gcp.extscloud.com:16552
topic: abc_abc-central-1.abc-tyur-service-in.v1
client_id: abc-tyur-service-dev-1
tls:
enabled: true
root_cas_file: /Users/ca.crt
client_certs:
- cert_file: /Users/cert.pem
key_file: /Users/key.pem
logger:
level: ALL
Thanks to benthos community on Discord