How to solve the xmlsec Error: (100, 'lxml & xmlsec libxml2 library version mismatch')

I am facing the following error when deploying my Django (version 4.1) Backend, I have the following Dockerfile (some non-relevant parts omitted) and need to install python3-saml (which has dependencies like lxml and xmlsec). The documentation mentions the following:

Hence I added the command in the Dockerfile.

FROM python:3.10.6

# Install necessary packages
RUN apt-get update && \
    apt-get install -y s3fs pkg-config libxml2-dev libxmlsec1-dev libxmlsec1-openssl libxmlsec1 && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*


# set work directory
WORKDIR /usr/src/app


# install dependencies
RUN pip install --upgrade pip
COPY ./requirements.txt .
RUN pip install -r requirements.txt

RUN pip install python3-saml && \
    pip install --force-reinstall --no-binary lxml lxml

# copy project
COPY . .

# create cache directory
RUN mkdir -p /var/tmp/django_cache

ENTRYPOINT ["./docker-entrypoint.sh"] 

I am able to build it without problem but upon deployment, I get the following error:

  File "/usr/local/lib/python3.10/site-packages/onelogin/saml2/auth.py", line 12, in <module>
    import xmlsec
xmlsec.Error: (100, 'lxml & xmlsec libxml2 library version mismatch')

I am not sure which library version is expected in this case, I have tried with pip install --no-binary lxml==4.6.3 lxml==4.6.3 --force-reinstall instead, as well as with version 4.9.3 (as seen in other recent threads) but with no success. (pip install python3-saml installs lxml version 5.2.2).

The requirement.txt file looks as follows:

apipkg==1.5
asgiref==3.7.2
asn1crypto==1.4.0
astroid==2.4.2
async-timeout==4.0.3
attrs==20.1.0
authy==2.2.6
autopep8==1.5.4
certifi==2020.6.20
cffi==1.15.1
chardet==3.0.4
click==7.1.2
colorama==0.4.6
coverage==5.2.1
cryptography==42.0.2
Cython==0.29.21
decorator==5.1.1
defusedxml==0.7.1
deprecation==2.1.0
dictor==0.1.11
Django==4.1
django-cors-headers==3.5.0
django-debug-toolbar==4.3.0
django-eav2==0.13.0
django-filter==2.4.0
django-liststyle==0.1
django-redis==5.4.0
django-redis-cache==3.0.1
django-rest-auth==0.9.5
django-reversion==5.0.3
django-simple-history==2.11.0
djangorestframework==3.14.0
djangorestframework-jwt==1.11.0
djangorestframework-simplejwt==5.2.2
drf-extensions==0.7.1
elementpath==4.2.1
exceptiongroup==1.2.0
execnet==1.7.1
factory-boy==3.0.1
Faker==4.1.2
flake8==3.8.3
future==0.18.2
idna==2.10
importlib-metadata==1.7.0
inflection==0.5.1
iniconfig==1.0.1
install==1.3.5
isodate==0.6.1
isort==5.5.3
Jinja2==2.11.2
lazy-object-proxy==1.4.3
MarkupSafe==1.1.1
mccabe==0.6.1
more-itertools==8.5.0
numpy==1.23.5
packaging==20.4
pandas==1.4.1
Paste==3.7.1
phonenumbers==8.12.16
pikepdf==7.2.0
Pillow==9.5.0
pluggy==0.13.1
psycopg2==2.9.3
psycopg2-binary==2.9.3
py==1.9.0
pycodestyle==2.6.0
pycparser==2.20
pycryptodomex==3.20.0
pyflakes==2.2.0
Pygments==2.6.1
PyJWT==1.7.1
pylint==2.6.0
pymemcache==4.0.0
PyMuPDF==1.22.3
pyOpenSSL==24.0.0
pyparsing==2.4.7
PyPDF2==1.26.0
pytest==7.1.3
pytest-cov==2.10.1
pytest-django==3.9.0
python-dateutil==2.8.1
python-memcached==1.59
pytz==2020.1
PyYAML==5.3.1
qrcode==7.3.1
redis==3.5.3
redispy==3.0.0
repoze.who==3.0.0
requests==2.24.0
six==1.15.0
slack-sdk==3.12.0
sqlparse==0.3.1
text-unidecode==1.3
toml==0.10.1
tomli==2.0.1
typing_extensions==4.9.0
tzdata==2023.4
urllib3==1.25.10
wcwidth==0.2.5
WebOb==1.8.7
Werkzeug==1.0.1
wrapt==1.12.1
xmlschema==2.5.1
zipp==3.1.0
zope.interface==6.1

If you updated your whole dependency tree, pip might be able to opt for newer xmlsec and lxml that were compiled for the same libxml2 version. pip install python3-saml in a new python:3.10 container works just fine.

Otherwise, I think you should compile both xmlsec and lxml. Ideally in a single install instruction so that pip does not accidentally re-install anything:

pip install --no-binary xmlsec,lxml python3-saml

You can even concat that in your requirements.txt so that everything is installed at once:

# requirements.txt
--no-binary xmlsec
--no-binary lxml
python3-saml
# ...
# ...

Otherwise the use of --force-reinstall without --no-deps can mangle whatever you thought you had already installed.