In my project's CI pipeline we build a docker image that contains a conda environment. Here is the Dockerfile:
FROM debian:buster-slim
RUN apt-get update && apt-get install curl gnupg -y && rm -rf /var/lib/apt/lists/*
SHELL ["/bin/bash", "-o", "pipefail", "-c"]
RUN curl https://repo.anaconda.com/pkgs/misc/gpgkeys/anaconda.asc | gpg --dearmor > conda.gpg && \
install -o root -g root -m 644 conda.gpg /usr/share/keyrings/conda-archive-keyring.gpg && \
gpg --keyring /usr/share/keyrings/conda-archive-keyring.gpg --no-default-keyring \
--fingerprint 34161F5BF5EB1D4BFBBB8F0A8AEB4F8B29D82806 && \
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/conda-archive-keyring.gpg] https://repo.anaconda.com/pkgs/misc/debrepo/conda stable main" \
> /etc/apt/sources.list.d/conda.list
WORKDIR /tmp
RUN MINICONDA_VERSION=4.10.3 && \
MINICONDA_SCRIPT="Miniconda3-py38_${MINICONDA_VERSION}-Linux-x86_64.sh" && \
CONDA_VERSION='4.10.3' && \
CONDA_DIR=/opt/conda && \
curl -O https://repo.anaconda.com/miniconda/${MINICONDA_SCRIPT} && \
/bin/bash ${MINICONDA_SCRIPT} -f -b -p $CONDA_DIR && \
rm ${MINICONDA_SCRIPT} && \
$CONDA_DIR/bin/conda config --system --set auto_update_conda false && \
$CONDA_DIR/bin/conda config --system --set show_channel_urls true && \
$CONDA_DIR/bin/conda config --system --remove channels defaults && \
$CONDA_DIR/bin/conda config --system --add channels main && \
$CONDA_DIR/bin/conda config --system --add channels conda-forge && \
$CONDA_DIR/bin/conda config --system --set env_prompt '({name}) ' && \
$CONDA_DIR/bin/conda config --system --append envs_dirs /opt/conda/envs/ && \
$CONDA_DIR/bin/conda config --system --append pkgs_dirs /opt/conda/pkgs/ && \
$CONDA_DIR/bin/conda config --system --append aggressive_update_packages conda-forge::google-* && \
$CONDA_DIR/bin/conda config --system --append aggressive_update_packages ca-certificates && \
$CONDA_DIR/bin/conda config --system --append aggressive_update_packages certifi && \
$CONDA_DIR/bin/conda config --system --append aggressive_update_packages openssl && \
$CONDA_DIR/bin/conda update --quiet --yes --all conda="${CONDA_VERSION}" && \
$CONDA_DIR/bin/conda create -n py3 python=3.8
RUN --mount=type=secret,id=pipconfig,dst=/etc/pip.conf \
bash -c "source /opt/conda/bin/activate /opt/conda/envs/py3 && conda install \
avro-python3 \
backports.zoneinfo \
behave \
black \
conda-tree \
dirty-equals \
flake8 \
google-api-python-client \
google-auth-oauthlib \
google-cloud-bigquery \
google-cloud-bigquery-storage \
google-cloud-bigtable \
google-cloud-datacatalog \
google-cloud-firestore \
google-cloud-logging \
google-cloud-monitoring \
google-cloud-pubsub \
google-cloud-secret-manager \
google-cloud-storage \
google-cloud-workflows \
invoke \
ipdb \
jinja2 \
jsonpath-ng \
oauth2client \
pandas \
pipdeptree \
pydantic \
pylint \
pytest \
pytest-xdist \
python-dateutil \
responses \
rich \
rope \
semantic_version \
sh \
sqlfluff \
tenacity \
tqdm \
zenpy \
&& pip install \
functions-framework \
google-cloud-bigquery==2.31.0 \
google-cloud-scheduler"
I wanted to update the conda environment to use python 3.10instead of python3.8 so I changed the appropriate line:
$CONDA_DIR/bin/conda create -n py3 python=3.10
then attempted to build the docker image. The building of the conda image failed with:
Found conflicts! Looking for incompatible packages.
This can take several minutes. Press CTRL-C to abort.
failed
UnsatisfiableError: The following specifications were found
to be incompatible with the existing python installation in your environment:
Specifications:
- backports.zoneinfo -> python[version='>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.9,<3.10.0a0|>=3.8,<3.9.0a0']
- google-cloud-bigquery -> python[version='2.7.*|3.5.*|3.6.*|>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.8,<3.9.0a0|>=3.7,<3.8.0a0']
- google-cloud-bigquery-storage -> python[version='>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0']
- oauth2client -> python[version='2.7.*|3.5.*|3.6.*|3.4.*']
- python-dateutil -> python[version='>=3.8,<3.9.0a0|>=3.9,<3.10.0a0']
- tenacity -> python[version='3.4.*|3.8.*|3.7.*']
Your python: python=3.10
If python is on the left-most side of the chain, that's the version you've asked for.
When python appears to the right, that indicates that the thing on the left is somehow
not available for the python version you are constrained to. Note that conda will not
change your python version to a different minor version unless you explicitly specify
that.
The significant line there seems to be:
When python appears to the right, that indicates that the thing on the left is somehow not available for the python version you are constrained to.
I've been digging in to try and discover the problem and focused on this line specifically:
google-cloud-bigquery -> python[version='2.7.|3.5.|3.6.*|>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.8,<3.9.0a0|>=3.7,<3.8.0a0']
The error suggests that google-cloud-bigquery is not available for python3.10 however I've checked at https://anaconda.org/conda-forge/google-cloud-bigquery which states that that package should be available for python3.10
Description google-cloud-bigquery installs google-cloud-bigquery-core and the extra requirements for pandas and pyarrow integrations.
Supported Python Versions Python >= 3.6
The last version of this library compatible with Python 2.7 and 3.5 is google-cloud-bigquery==1.28.0.
On the surface it seems to me that the package I'm trying to install is available for python3.10 so I don't understand why I'm getting this conflict. Clearly I'm missing some important information but I'm not familiar enough with conda to understand what. Any advice much appreciated.
I solved this by removing backports.zoneinfo (which isn't required in python3.9 and above because its baked into the standard library) and google-cloud-bigquery from the conda installs. google-cloud-bigquery was being being installed by pip anyway (as you can see in my original post on this thread) because conda was installing an earlier version to the one we desired (I don't know why).
I also removed tqdm because we are not using it any more. The resulting Dockerfile looks like this:
FROM debian:buster-slim
RUN apt-get update && apt-get install curl gnupg -y && rm -rf /var/lib/apt/lists/*
SHELL ["/bin/bash", "-o", "pipefail", "-c"]
RUN curl https://repo.anaconda.com/pkgs/misc/gpgkeys/anaconda.asc | gpg --dearmor > conda.gpg && \
install -o root -g root -m 644 conda.gpg /usr/share/keyrings/conda-archive-keyring.gpg && \
gpg --keyring /usr/share/keyrings/conda-archive-keyring.gpg --no-default-keyring \
--fingerprint 34161F5BF5EB1D4BFBBB8F0A8AEB4F8B29D82806 && \
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/conda-archive-keyring.gpg] https://repo.anaconda.com/pkgs/misc/debrepo/conda stable main" \
> /etc/apt/sources.list.d/conda.list
WORKDIR /tmp
RUN MINICONDA_VERSION=4.10.3 && \
MINICONDA_SCRIPT="Miniconda3-py38_${MINICONDA_VERSION}-Linux-x86_64.sh" && \
CONDA_VERSION='4.10.3' && \
CONDA_DIR=/opt/conda && \
curl -O https://repo.anaconda.com/miniconda/${MINICONDA_SCRIPT} && \
/bin/bash ${MINICONDA_SCRIPT} -f -b -p $CONDA_DIR && \
rm ${MINICONDA_SCRIPT} && \
$CONDA_DIR/bin/conda config --system --set auto_update_conda false && \
$CONDA_DIR/bin/conda config --system --set show_channel_urls true && \
$CONDA_DIR/bin/conda config --system --remove channels defaults && \
$CONDA_DIR/bin/conda config --system --add channels main && \
$CONDA_DIR/bin/conda config --system --add channels conda-forge && \
$CONDA_DIR/bin/conda config --system --set env_prompt '({name}) ' && \
$CONDA_DIR/bin/conda config --system --append envs_dirs /opt/conda/envs/ && \
$CONDA_DIR/bin/conda config --system --append pkgs_dirs /opt/conda/pkgs/ && \
$CONDA_DIR/bin/conda config --system --append aggressive_update_packages conda-forge::google-* && \
$CONDA_DIR/bin/conda config --system --append aggressive_update_packages ca-certificates && \
$CONDA_DIR/bin/conda config --system --append aggressive_update_packages certifi && \
$CONDA_DIR/bin/conda config --system --append aggressive_update_packages openssl && \
$CONDA_DIR/bin/conda update --quiet --yes --all conda="${CONDA_VERSION}" && \
$CONDA_DIR/bin/conda create -n py3 python=3.10
RUN --mount=type=secret,id=pipconfig,dst=/etc/pip.conf \
bash -c "source /opt/conda/bin/activate /opt/conda/envs/py3 && conda install \
avro-python3 \
behave \
black \
conda-tree \
dirty-equals \
flake8 \
google-api-python-client \
google-auth-oauthlib \
google-cloud-bigtable \
google-cloud-datacatalog \
google-cloud-firestore \
google-cloud-logging \
google-cloud-monitoring \
google-cloud-pubsub \
google-cloud-secret-manager \
google-cloud-storage \
google-cloud-workflows \
invoke \
ipdb \
jinja2 \
jsonpath-ng \
oauth2client \
pandas \
pipdeptree \
pydantic \
pylint \
pytest \
pytest-xdist \
python-dateutil \
responses \
rich \
rope \
semantic_version \
sh \
sqlfluff \
tenacity \
zenpy \
&& pip install \
functions-framework \
google-cloud-bigquery \
google-cloud-scheduler \
object-config-models"