I am uncertain of how to do development against an existing connector using pyairbyte. I cannot get pyairbyte to successfully load my modified source connector. I have made changes to the source-appsflyer connector on my local clone of the linked repo. I am trying to test my changes using pyairbyte with simple code below
import airbyte as ab
import json
CONFIG_PATH = "configs/appsflyer.json"
with open(CONFIG_PATH, "r") as f:
source_config = json.load(f)
## /app/source_appsflyer/ is my local clone of airbytehq/airbyte/ mounting only the relevant appsflyer connector folder in my docker container.
source = ab.get_source("source-appsflyer", config=source_config, local_executable="/root/.cache/pypoetry/virtualenvs/source-appsflyer-OcVLBknA-py3.10/bin/source-appsflyer")
source.select_streams(["my_target_stream"])
all_streams = source.get_selected_streams()
read_result = source.read()
Following the instructions in the connector's readme I tried poetry install --with dev which places an executable in /root/.cache/pypoetry/virtualenvs/source-appsflyer-OcVLBknA-py3.10/bin/source-appsflyer, which I point to above with the local_executable param. This leads to working code. However, that is because it appears to install directly from PyPI, ignoring my local changes. I validated this by making changes to the source code and then doing a fresh install. I then check for those changes to appear in the lib code under /root/.cache/pypoetry/virtualenvs/source-appsflyer-OcVLBknA-py3.10/lib/.
I then tried to test this by installing the source-appsflyer connector with pip install into my local environment and then modifying the above to point at the entrypoint for the package with source = ab.get_source("source-appsflyer", config=source_config, local_executable="/app/source_appsflyer/source_appsflyer/run.py") but this fails because I get package name collisions for import airbyte between airebyte-cdk and pyairbyte.
Finally, I tried poetry build from the source-appsflyer source to build a wheel of my local changes, and then pointing local_executable at that wheel with source = ab.get_source("source-appsflyer", config=source_config, local_executable="/app/source_appsflyer/dist/name_of_the_whl_here") This fails.
What is the proper way to do this? How can I point to an local executable or path that pyairbyte understands?
I have "solved" this in a very unsatisfactory way with the following:
with poetry installed, I first ensure that /app/source_appsflyer/ has a poetry virtenv with poetry env activate
I then build a wheel from that directory with poetry build
I then modify the requirements.txt of source_appsflyer , modifying the package from airbyte-source-appsflyer with no pinning, to source-appsflyer @ file:///app/source_appsflyer/dist/source_appsflyer-0.2.40-py3-none-any.whl. Modifying the name to match the package name in source code contained in the wheel (otherwise, pip throws an error).
I then then install the wheel locally with pip install -r requirements.txt ensuring that I am using the pip executable from the activated poetry environment.
At this point, I can now specify the local executable for my source in my test script above with
source = ab.get_source("source-appsflyer", config=source_config, local_executable="/root/.cache/pypoetry/virtualenvs/source-appsflyer-OcVLBknA-py3.10/bin/source-appsflyer")
If I need to make any changes, I rebuild the wheel, and reinstall with pip install -r requirements.txt --force-reinstall
There is probably a less convoluted way of doing this, but I am unblocked with this flow. Any feedback on improving this flow is welcome.