[SOLVED] Scrapy and Great Expectations (great_expectations)

Scrapy and Great Expectations (great_expectations) - Not working together

I am trying to use the packages scrapy and great_expectations within the same virtual environment. There seems to be an issue with the compatibility between the two packages, depending on the order in which I import them in.

Example:

I created a virtual environment and pip installed the latest version of each package.

Does work:

import great_expectations
import scrapy

print("done")

Does not work:

import scrapy
import great_expectations

print("done")

Error:

Traceback (most recent call last):
  File 

"/Users/grant/vs_code_projects/grants_projects/test_environment.py", line 2, in <module>
    import great_expectations
  File "/Users/grant/Envs/test_env/lib/python3.8/site-packages/great_expectations/__init__.py", line 32, in <module>
    register_core_expectations()
  File "/Users/grant/Envs/test_env/lib/python3.8/site-packages/great_expectations/expectations/registry.py", line 187, in register_core_expectations
    from great_expectations.expectations import core  # noqa: F401
  File "/Users/grant/Envs/test_env/lib/python3.8/site-packages/great_expectations/expectations/core/__init__.py", line 1, in <module>
    from .expect_column_distinct_values_to_be_in_set import (
  File "/Users/grant/Envs/test_env/lib/python3.8/site-packages/great_expectations/expectations/core/expect_column_distinct_values_to_be_in_set.py", line 12, in <module>
    from great_expectations.expectations.expectation import (
  File "/Users/grant/Envs/test_env/lib/python3.8/site-packages/great_expectations/expectations/expectation.py", line 2350, in <module>
    class BatchExpectation(Expectation, ABC):
  File "/Users/grant/Envs/test_env/lib/python3.8/site-packages/great_expectations/expectations/expectation.py", line 287, in __new__
    newclass._register_renderer_functions()
  File "/Users/grant/Envs/test_env/lib/python3.8/site-packages/great_expectations/expectations/expectation.py", line 369, in _register_renderer_functions
    attr_obj: Callable = getattr(cls, candidate_renderer_fn_name)
AttributeError: __provides__

Edit:

I have tried all major versions of Scrapy (current version, 2.0.0, 1.0.0) with all major versions of Great Expectations and it is the same result.

Solution