SystemExit: error: google-auth 2.0.0.dev0 is installed but google-auth<2.0dev,>=1.24.0 is required by {'google-cloud-core'}
This error appears now on AWS Glue Jobs, in my case, jobs using smart_open. Did not happen until now and I was already using .eggs on AWS Glue to be able to run smart_open.
There is a new version of google-auth 1.33.0 since 20th of July 2021, those anyone else encountered this error? AWS has to update this error?
Maybe using a .egg file of google-auth 1.33.0 will solve that issue. Refer to these post on how to do an .egg file How to create Python egg file
I was already using an egg for my pythonshell jobs. However I was not freezing the version numbers.
setup.py
from setuptools import setup
setup(
name="my_glue_dependencies",
install_requires=[
'google-auth==1.33.1',
'google-cloud-storage==1.41.1',
'google-cloud-bigquery==2.22.0'
]
)
The key to freezing the versions used is to use == instead of >= for the install_requires options.
The docker image from this article can be used to build the egg.
Build the egg $ python setup.py bdist_egg -d ./dist
The full article on python glue jobs, and how to Provide Your Own Python Library.