pythonpython-packagingpython-poetryairbytepyairbyte

Pyairbyte will not pick up my local changes to a source connector


I am uncertain of how to do development against an existing connector using pyairbyte. I cannot get pyairbyte to successfully load my modified source connector. I have made changes to the source-appsflyer connector on my local clone of the linked repo. I am trying to test my changes using pyairbyte with simple code below

import airbyte as ab
import json

CONFIG_PATH = "configs/appsflyer.json"
with open(CONFIG_PATH, "r") as f:
    source_config = json.load(f)

## /app/source_appsflyer/ is my local clone of airbytehq/airbyte/ mounting only the relevant appsflyer connector folder in my docker container.

source = ab.get_source("source-appsflyer", config=source_config, local_executable="/root/.cache/pypoetry/virtualenvs/source-appsflyer-OcVLBknA-py3.10/bin/source-appsflyer")
source.select_streams(["my_target_stream"])
all_streams = source.get_selected_streams()
read_result = source.read()

Following the instructions in the connector's readme I tried poetry install --with dev which places an executable in /root/.cache/pypoetry/virtualenvs/source-appsflyer-OcVLBknA-py3.10/bin/source-appsflyer, which I point to above with the local_executable param. This leads to working code. However, that is because it appears to install directly from PyPI, ignoring my local changes. I validated this by making changes to the source code and then doing a fresh install. I then check for those changes to appear in the lib code under /root/.cache/pypoetry/virtualenvs/source-appsflyer-OcVLBknA-py3.10/lib/.

I then tried to test this by installing the source-appsflyer connector with pip install into my local environment and then modifying the above to point at the entrypoint for the package with source = ab.get_source("source-appsflyer", config=source_config, local_executable="/app/source_appsflyer/source_appsflyer/run.py") but this fails because I get package name collisions for import airbyte between airebyte-cdk and pyairbyte.

Finally, I tried poetry build from the source-appsflyer source to build a wheel of my local changes, and then pointing local_executable at that wheel with source = ab.get_source("source-appsflyer", config=source_config, local_executable="/app/source_appsflyer/dist/name_of_the_whl_here") This fails.

What is the proper way to do this? How can I point to an local executable or path that pyairbyte understands?


Solution

  • I have "solved" this in a very unsatisfactory way with the following:

    There is probably a less convoluted way of doing this, but I am unblocked with this flow. Any feedback on improving this flow is welcome.