I'm setting up a custom connector in an opensource installation of airbyte, to get data from an api. The problem is that the api's ssl certificate is self signed and airbyte is complaining about it.
Is there a way to disable ssl verification? ps: I'm building from the UI
As of now, I have not found a way to do this from the Builder in the UI. We have written a custom connector for this purpose. Use the cdk to create a custom HTTP connector. Create a stream class (see below). We have added the code to send the request in the parse_response function (but not sure that this is the right place, but it works). We use the requests package from python to send the request to the endpoint. In Python request call you set verify to False, which means it ignores the SSL verification. This is an example using basic authentication:
class AStream(HttpStream):
__username = ""
__password = ""
url_base = ""
primary_key = None
def __init__(self, config: Mapping[str,Any], **kwargs):
super().__init__()
# These values come from the UI
self.__username = config['username']
self.__password = config['password']
self.url_base = "https://" + config['server']
def parse_response(
self,
response: requests.Response,
stream_state: Mapping[str, Any],
stream_slice: Mapping[str, Any] = None,
next_page_token: Mapping[str, Any] = None,
) -> Iterable[Mapping]:
url = self.url_base + <path to endpoint>
response = requests.get(url, verify=False, auth=HTTPBasicAuth(self.__username, self.__password))
return response.json() # you can also iterate over your received object and use yield to return multiple objects