I am uncertain of how to do development against an existing connector using pyairbyte. I cannot get pyairbyte
to successfully load my modified source connector. I have made changes to the source-appsflyer connector on my local clone of the linked repo. I am trying to test my changes using pyairbyte with simple code below
import airbyte as ab
import json
CONFIG_PATH = "configs/appsflyer.json"
with open(CONFIG_PATH, "r") as f:
source_config = json.load(f)
## /app/source_appsflyer/ is my local clone of airbytehq/airbyte/ mounting only the relevant appsflyer connector folder in my docker container.
source = ab.get_source("source-appsflyer", config=source_config, local_executable="/root/.cache/pypoetry/virtualenvs/source-appsflyer-OcVLBknA-py3.10/bin/source-appsflyer")
source.select_streams(["my_target_stream"])
all_streams = source.get_selected_streams()
read_result = source.read()
Following the instructions in the connector's readme I tried poetry install --with dev
which places an executable in /root/.cache/pypoetry/virtualenvs/source-appsflyer-OcVLBknA-py3.10/bin/source-appsflyer
, which I point to above with the local_executable
param. This leads to working code. However, that is because it appears to install directly from PyPI, ignoring my local changes. I validated this by making changes to the source code and then doing a fresh install. I then check for those changes to appear in the lib
code under /root/.cache/pypoetry/virtualenvs/source-appsflyer-OcVLBknA-py3.10/lib/
.
I then tried to test this by installing the source-appsflyer
connector with pip install
into my local environment and then modifying the above to point at the entrypoint for the package with source = ab.get_source("source-appsflyer", config=source_config, local_executable="/app/source_appsflyer/source_appsflyer/run.py")
but this fails because I get package name collisions for import airbyte between airebyte-cdk and pyairbyte.
Finally, I tried poetry build
from the source-appsflyer
source to build a wheel of my local changes, and then pointing local_executable
at that wheel with source = ab.get_source("source-appsflyer", config=source_config, local_executable="/app/source_appsflyer/dist/name_of_the_whl_here")
This fails.
What is the proper way to do this? How can I point to an local executable or path that pyairbyte understands?
I have "solved" this in a very unsatisfactory way with the following:
with poetry installed, I first ensure that /app/source_appsflyer/
has a poetry virtenv with poetry env activate
I then build a wheel from that directory with poetry build
I then modify the requirements.txt
of source_appsflyer
, modifying the package from airbyte-source-appsflyer
with no pinning, to source-appsflyer @ file:///app/source_appsflyer/dist/source_appsflyer-0.2.40-py3-none-any.whl
. Modifying the name to match the package name in source code contained in the wheel (otherwise, pip throws an error).
I then then install the wheel locally with pip install -r requirements.txt
ensuring that I am using the pip
executable from the activated poetry environment.
At this point, I can now specify the local executable for my source in my test script above with
source = ab.get_source("source-appsflyer", config=source_config, local_executable="/root/.cache/pypoetry/virtualenvs/source-appsflyer-OcVLBknA-py3.10/bin/source-appsflyer")
If I need to make any changes, I rebuild the wheel, and reinstall with pip install -r requirements.txt --force-reinstall
There is probably a less convoluted way of doing this, but I am unblocked with this flow. Any feedback on improving this flow is welcome.