I don't have a lot of experience with python project organization, but I've been trying to follow modern best practices using a pyproject.toml and structure as in this video (minus the ci bits).
This is ultimately a geoprocessing tool for ArcGIS Pro, though I've tried to abstract the logic of the tool from the 'backend' calls to ArcGIS Pro so that I could create other backends in the future, for example for QGIS or geopandas and rasterio.
However, the arcpy module that allows one to interface with ArcGIS Pro is not a module you can just install into your environment from pypi. It simply "exists" in the conda environment provided by the install of ArcGIS Pro. I've been writing and running scripts by activating this conda env. I don't think it would work to list arcpy as a dependency in pyproject.toml since it can't be installed.
How do you deal with this situation? What are best practices?
Note: The arcpy package might be installable via conda, according to this doc from esri. At the very least, I can clone the default env provided by ArcGIS Pro and modify that (to install dev deps). As stated, I'm quite new to python project management (and it's rather confusing), but if I listed my deps in pyproject.toml, generated a requirements.txt, then installed them into a conda dev env from requirements.txt, might that work?
pip can see arcpy in the conda env:
PS> pip list
Package Version
--------------------------------- -----------------
anyio 3.5.0
appdirs 1.4.4
arcgis 2.2.0.1
...
How do you deal with this situation? What are best practices?
So you should list arcpy in your pyproject.toml or requirements.
It simply "exists" in the conda environment provided by the install of ArcGIS Pro
Great. So if you list the dependency, users will know that you require the ArcGIS Pro to work.
I've tried to abstract the logic of the tool from the 'backend' calls to ArcGIS Pro so that I could create other backends in the future, for example for QGIS or geopandas and rasterio.
Great, so you are making two packages. There is "backend" which implements some interface on top of ArcGIS Pro. That interface is then registered in what you would call "frontend" that does not depend on ArcGIS Pro. These two packages have separate requirements.
To give a real life example, pyscada project has https://github.com/pyscada/PyScada/blob/main/setup.py#L33 and then has plugins with separate https://github.com/pyscada/PyScada-BACnet/blob/main/setup.py#L33 dependencies.
if I listed my deps in pyproject.toml, generated a requirements.txt, then installed them into a conda dev env from requirements.txt, might that work?
Yes.
You either list the dependencies in pyproject.toml like https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#a-full-example or write requirements.txt, which you can then load dynamically from pyproject.toml like How to reference a requirements.txt in the pyproject.toml of a setuptools project? .