apache-kafkaapache-kafka-connectkafka-rest

Kafka REST API source connector with authentication header


I need to create kafka source connector for REST API with Header authentication like curl -H "Authorization: Basic " -H "clientID: " "https:< url for source> " . I am using apache kafka , I used connector class com.github.castorm.kafka.connect.http.HttpSourceConnector Here is my json file for connector

{
    "name": "rest_data6",
    "config": {
           "key.converter":"org.apache.kafka.connect.json.JsonConverter",
           "value.converter":"org.apache.kafka.connect.json.JsonConverter",
           "key.converter.schemas.enable":"true",
           "value.converter.schemas.enable":"true",
            "connector.class": "com.github.castorm.kafka.connect.http.HttpSourceConnector",
           "tasks.max": "1",
            "http.request.headers": "Authorization: Basic <key1>",
            "http.request.headers": "clientID: <key>",
             "http.request.url": "https:<url for source ?",
           "kafka.topic": "mysqltopic2"
    }
}

Also I tried with "connector.class": "com.tm.kafka.connect.rest.RestSourceConnector", My joson file as below

"name": "rest_data2",
"config": {

       "key.converter":"org.apache.kafka.connect.json.JsonConverter",
       "value.converter":"org.apache.kafka.connect.json.JsonConverter",
       "key.converter.schemas.enable":"true",
       "value.converter.schemas.enable":"true",
       "connector.class": "com.tm.kafka.connect.rest.RestSourceConnector",
       "rest.source.poll.interval.ms": "900",
       "rest.source.method": "GET",
       "rest.source.url":"URL of source ",
       "tasks.max": "1",
       "rest.source.headers": "Authorization: Basic <key> , clientId :<key2>",
"rest.source.topic.selector": "com.tm.kafka.connect.rest.selector.SimpleTopicSelector",
"rest.source.destination.topics":  "mysql1"

}

}

But no hope . Any idea how to GET REST API data with authentication . My authentication parameter is Authorization: Basic and Authorization: Basic .

Just for mention both the file are working with REST API without authentication , once I added authentication parameter then wither connector status is failed or It produce ":"Cannot route. Codebase/company is invalid"" message in topic. Can any one suggest what is way to solve it


Solution

  • I mailed to original developer to Cástor Rodríguez. As per his solution I modified my json

    Put header into a single form and it works

    {
        "name": "rest_data6",
        "config": {
               "key.converter":"org.apache.kafka.connect.json.JsonConverter",
               "value.converter":"org.apache.kafka.connect.json.JsonConverter",
               "key.converter.schemas.enable":"true",
               "value.converter.schemas.enable":"true",
                "connector.class": "com.github.castorm.kafka.connect.http.HttpSourceConnector",
               "tasks.max": "1",
                "http.request.headers": "Authorization: Basic <key1>, clientID: <key>"
                 "http.request.url": "https:<url for source ?",
               "kafka.topic": "mysqltopic2"
        }
    }
    

    enter image description here