There seems to be an issue logged with the current Confluent-Kafka package on pypi:
I have a Dockerfile with the foll code which used to work until the issue happened :
RUN cd /tmp && git clone https://github.com/edenhill/librdkafka.git \
&& cd librdkafka \
&& ./configure --prefix /usr \
&& make \
&& make install \
&& cd .. \
&& rm -rf librdkafka
RUN apk --update add \
......
pip3 install -U setuptools &&\
pip3 install --upgrade pip \
avro-python3\
confluent-kafka[avro]\
confluent-kafka\
ERROR after the issue: Command errored out with exit status 1:
command: /usr/bin/python3.6 -c 'import sys, setuptools, tokenize; sys.argv[0] =
'"'"'/tmp/pip-req-build-14u6lptl/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-
14u6lptl/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)
(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code,
__file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-req-build-14u6lptl/pip-egg-info
cwd: /tmp/pip-req-build-14u6lptl/
Complete output (5 lines):
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-req-build-14u6lptl/setup.py", line 12, in <module>
with open(os.path.join(mod_dir, 'requirements.txt')) as f:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-req-build-
14u6lptl/confluent_kafka/requirements.txt'
on the github page for the issue, there were some suggestions made for work arounds the issue, which I have tried like so:
1) workaround which didnt work:
ENV LIBRDKAFKA_VERSION v1.4.0 \
RUN cd /tmp && git clone https://github.com/edenhill/librdkafka.git \
&& cd librdkafka \
&& git checkout $LIBRDKAFKA_VERSION \
&& ./configure --install-deps --prefix /usr \
&& make \
&& make install
RUN apk --update add \
......
pip3 install -U setuptools &&\
pip3 install --upgrade pip \
avro-python3\
confluent-kafka[avro]\
confluent-kafka\
....
2) workaround which didnt work:
ENV LIBRDKAFKA_VERSION v1.4.0 \
RUN cd /tmp && git clone https://github.com/edenhill/librdkafka.git \
&& cd librdkafka \
&& git checkout $LIBRDKAFKA_VERSION \
&& ./configure --install-deps --prefix /usr \
&& make \
&& make install
RUN apk --update add \
.....
pip3 install -U setuptools &&\
pip3 install --upgrade pip \
avro-python3\
confluent-kafka[avro]\
--no-binary :all: -i https://test.pypi.org/simple/ confluent-kafka==v1.4.0.1 \
....
ERROR:
Service 'xxxx' failed to build: The command '/bin/sh -c apk --update add
avro-python3
confluent-kafka[avro]
--no-binary :all: -i https://test.pypi.org/simple/ confluent-kafka==v1.4.0.1
returned a non-zero code: 1
Im wondering if there is a workaround I havent tried that someone can suggest to get me going until they fix this issue..
Just pin to an old version until they make a new release:
pip3 install --upgrade pip \
avro-python3\
confluent-kafka[avro]==1.3.0\
Also note that installing both the main package (confluent-kafka) and the package with the extra (confluent-kafka[avro]) is unnecessary, you just need to install the package with the extra. You're essentially installing it twice.