this is my first entry and I hope you can help me:
My plan is to build a docker image for an armv7 architecture linux system based on the "python:3.7-buster" image. My problem is that I'm not able to install the google-cloud-bigquery python package during the imagebuild.
I'm building the image on my Windows 10 system via:
docker buildx build --platform=linux/arm/v7 .
I'm trying to install google-cloud-bigquery python package in my Dockerfile trying to install it in a virtualenv like it's recommended here .
So I insertet this into my Dockerfile:
...
RUN pip install virtualenv
RUN virtualenv env
RUN . env/bin/activate
RUN env/bin/python -m pip install --upgrade pip
RUN env/bin/pip install google-cloud-bigquery
...
But that gives me this ERROR logs:
...
------
> [14/18] RUN env/bin/pip install google-cloud-bigquery:
#17 9.450 Collecting google-cloud-bigquery
#17 9.716 Downloading google_cloud_bigquery-2.26.0-py2.py3-none-any.whl (201 kB)
#17 11.07 Collecting proto-plus>=1.10.0
#17 11.13 Downloading proto_plus-1.19.0-py3-none-any.whl (42 kB)
#17 11.99 Collecting google-resumable-media<3.0dev,>=0.6.0
#17 12.05 Downloading google_resumable_media-2.0.1-py2.py3-none-any.whl (75 kB)
#17 12.96 Collecting packaging>=14.3
#17 13.02 Downloading packaging-21.0-py3-none-any.whl (40 kB)
#17 14.65 Collecting google-api-core[grpc]<3.0.0dev,>=1.29.0
#17 14.71 Downloading google_api_core-2.0.1-py2.py3-none-any.whl (92 kB)
#17 16.23 Collecting requests<3.0.0dev,>=2.18.0
#17 16.29 Downloading requests-2.26.0-py2.py3-none-any.whl (62 kB)
#17 17.27 Collecting google-cloud-core<3.0.0dev,>=1.4.1
#17 17.33 Downloading google_cloud_core-2.0.0-py2.py3-none-any.whl (27 kB)
#17 23.43 Collecting protobuf>=3.12.0
#17 23.49 Downloading protobuf-3.17.3-py2.py3-none-any.whl (173 kB)
#17 43.54 Collecting grpcio<2.0dev,>=1.38.1
#17 43.61 Downloading grpcio-1.39.0-cp37-cp37m-linux_armv7l.whl (37.8 MB)
#17 59.51 Requirement already satisfied: setuptools>=40.3.0 in ./env/lib/python3.7/site-packages (from google-api-core[grpc]<3.0.0dev,>=1.29.0->google-cloud-bigquery) (57.4.0)
#17 59.96 Collecting googleapis-common-protos<2.0dev,>=1.6.0
#17 60.03 Downloading googleapis_common_protos-1.53.0-py2.py3-none-any.whl (198 kB)
#17 62.65 Collecting google-auth<3.0dev,>=1.25.0
#17 62.72 Downloading google_auth-2.0.2-py2.py3-none-any.whl (152 kB)
#17 64.70 Collecting pyasn1-modules>=0.2.1
#17 64.76 Downloading pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
#17 65.60 Collecting rsa<5,>=3.1.4
#17 65.66 Downloading rsa-4.7.2-py3-none-any.whl (34 kB)
#17 66.36 Collecting cachetools<5.0,>=2.0.0
#17 66.43 Downloading cachetools-4.2.2-py3-none-any.whl (11 kB)
#17 68.21 Collecting google-crc32c<2.0dev,>=1.0
#17 68.26 Downloading google-crc32c-1.1.3.tar.gz (12 kB)
#17 70.61 Installing build dependencies: started
#17 106.1 Installing build dependencies: finished with status 'done'
#17 106.2 Getting requirements to build wheel: started
#17 111.7 Getting requirements to build wheel: finished with status 'done'
#17 111.7 Preparing wheel metadata: started
#17 115.9 Preparing wheel metadata: finished with status 'done'
#17 116.7 Collecting six>=1.5.2
#17 116.7 Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
#17 118.2 Collecting pyparsing>=2.0.2
#17 118.3 Downloading pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
#17 120.2 Collecting pyasn1<0.5.0,>=0.4.6
#17 120.3 Downloading pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
#17 121.5 Collecting certifi>=2017.4.17
#17 121.6 Downloading certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
#17 122.1 Collecting idna<4,>=2.5
#17 122.2 Downloading idna-3.2-py3-none-any.whl (59 kB)
#17 122.8 Collecting charset-normalizer~=2.0.0
#17 122.9 Downloading charset_normalizer-2.0.4-py3-none-any.whl (36 kB)
#17 124.2 Collecting urllib3<1.27,>=1.21.1
#17 124.2 Downloading urllib3-1.26.6-py2.py3-none-any.whl (138 kB)
#17 126.3 Building wheels for collected packages: google-crc32c
#17 126.4 Building wheel for google-crc32c (PEP 517): started
#17 132.6 Building wheel for google-crc32c (PEP 517): finished with status 'error'
#17 132.6 ERROR: Command errored out with exit status 1:
#17 132.6 command: /usr/src/app/env/bin/python /usr/src/app/env/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /tmp/tmpklqoljns
#17 132.6 cwd: /tmp/pip-install-px6rjego/google-crc32c_45773aee0be643769a80777ab94173a8
#17 132.6 Complete output (22 lines):
#17 132.6 running bdist_wheel
#17 132.6 running build
#17 132.6 running build_py
#17 132.6 creating build
#17 132.6 creating build/lib.linux-armv7l-3.7
#17 132.6 creating build/lib.linux-armv7l-3.7/google_crc32c
#17 132.6 copying src/google_crc32c/python.py -> build/lib.linux-armv7l-3.7/google_crc32c
#17 132.6 copying src/google_crc32c/__config__.py -> build/lib.linux-armv7l-3.7/google_crc32c
#17 132.6 copying src/google_crc32c/cext.py -> build/lib.linux-armv7l-3.7/google_crc32c
#17 132.6 copying src/google_crc32c/_checksum.py -> build/lib.linux-armv7l-3.7/google_crc32c
#17 132.6 copying src/google_crc32c/__init__.py -> build/lib.linux-armv7l-3.7/google_crc32c
#17 132.6 running build_ext
#17 132.6 building 'google_crc32c._crc32c' extension
#17 132.6 creating build/temp.linux-armv7l-3.7
#17 132.6 creating build/temp.linux-armv7l-3.7/src
#17 132.6 creating build/temp.linux-armv7l-3.7/src/google_crc32c
#17 132.6 gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/src/app/env/include -I/usr/local/include/python3.7m -c src/google_crc32c/_crc32c.c -o build/temp.linux-armv7l-3.7/src/google_crc32c/_crc32c.o
#17 132.6 src/google_crc32c/_crc32c.c:3:10: fatal error: crc32c/crc32c.h: No such file or directory
#17 132.6 #include <crc32c/crc32c.h>
#17 132.6 ^~~~~~~~~~~~~~~~~
#17 132.6 compilation terminated.
#17 132.6 error: command 'gcc' failed with exit status 1
#17 132.6 ----------------------------------------
#17 132.6 ERROR: Failed building wheel for google-crc32c
#17 132.6 Failed to build google-crc32c
#17 132.6 ERROR: Could not build wheels for google-crc32c which use PEP 517 and cannot be installed directly
------
error: failed to solve: executor failed running [/bin/sh -c env/bin/pip install google-cloud-bigquery]: exit code: 1
So I searched for this error and found this entry. I tried to install python3-dev to get the required python development headers with this:
...
RUN apt-get install python3-dev -y
RUN pip install virtualenv
RUN virtualenv env
RUN . env/bin/activate
RUN env/bin/python -m pip install --upgrade pip
RUN env/bin/pip install google-cloud-bigquery
...
But this gives me the same error like above.
So I thought I have to install it inside the virtualenv. "apt-get" didn't woked for me so I tried it with "pip3" in my Dockerfile:
...
RUN pip install virtualenv
RUN virtualenv env
RUN . env/bin/activate
RUN env/bin/python -m pip install --upgrade pip
RUN env/bin/pip3 install python3-dev
RUN env/bin/pip install google-cloud-bigquery
...
And this gave me this error logs:
...
=> ERROR [11/16] RUN env/bin/pip3 install python3-dev 8.4s
------
> [11/16] RUN env/bin/pip3 install python3-dev:
#15 7.857 ERROR: Could not find a version that satisfies the requirement python3-dev (from versions: none)
#15 7.860 ERROR: No matching distribution found for python3-dev
------
error: failed to solve: executor failed running [/bin/sh -c env/bin/pip3 install python3-dev]: exit code: 1
I also tried steps on this site. I installed gcc, build-essential and libxslt-dev, libffi-dev and libssl-dev. And I tried to install "python3-dev_3.7.3-1_armhf.deb" manually from this page.
Nothing worked. So how can I install the google-cloud-bigquery python package?
Here is my whole Dockerfile:
FROM python:3.7-buster
WORKDIR /usr/src/app
COPY python3-dev_3.7.3-1_armhf.deb ./
RUN pip install --upgrade pip
RUN apt-get update
RUN apt-get upgrade -y
RUN apt-get install gcc
RUN apt-get install build-essential
RUN apt-get install python3-dev -y
RUN apt-get install libxslt-dev libffi-dev libssl-dev
RUN apt-get install ./python3-dev_3.7.3-1_armhf.deb
RUN pip install virtualenv
RUN virtualenv env
RUN . env/bin/activate
RUN env/bin/python -m pip install --upgrade pip
RUN env/bin/pip install google-cloud-bigquery
Thank you for any help!
Installing google-crc32c==1.1.2 in my Dockerfile solved the problem! So this is my final Dockefile:
FROM python:3.7-buster
WORKDIR /usr/src/app
RUN pip install --upgrade pip
RUN apt-get update
RUN apt-get upgrade -y
RUN pip install virtualenv
RUN virtualenv env
RUN . env/bin/activate
RUN env/bin/python -m pip install --upgrade pip
RUN env/bin/pip install "google-crc32c==1.1.2"
RUN env/bin/pip install google-cloud-bigquery
Stumbled upon the same error right now, trying to install google-cloud-storage using poetry.
For now, I went around it by installing the 1.1.2 version of the google-crc32c package (not the latest one).
poetry add "google-crc32c=1.1.2"
I did it using poetry, but pip should work as well.
The error is gone on my side. Maybe this helps.