pythonpandasdataframeraymodin

Pandas Modin ray library fails to startup

I am trying to accelerate my pandas data processing using modin

import os
os.environ["MODIN_ENGINE"] = "ray"
import modin.pandas as pd

df = pd.read_csv(r"C:\Users\Harshad\Documents\Files\Data\Pre-processed\data.csv", low_memory=False)

I get the below warnings and error:

UserWarning: Ray execution environment not yet initialized. Initializing...
To remove this warning, run the following python code before doing dataframe operations:

    import ray
    ray.init()

Traceback (most recent call last):
  File "C:\Users\Harshad\Documents\pythonProject\venv\lib\site-packages\ray\node.py", line 240, in __init__
    self.redis_password)
  File "C:\Users\Harshad\Documents\pythonProject\venv\lib\site-packages\ray\_private\services.py", line 328, in wait_for_node
    raise TimeoutError("Timed out while waiting for node to startup.")
TimeoutError: Timed out while waiting for node to startup.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Users/Harshad/Documents/Code/data.py", line 18, in <module>
    low_memory=False)
  File "C:\Users\Harshad\Documents\pythonProject\venv\lib\site-packages\modin\pandas\io.py", line 135, in read_csv
    return _read(**kwargs)
  File "C:\Users\Harshad\Documents\pythonProject\venv\lib\site-packages\modin\pandas\io.py", line 58, in _read
    Engine.subscribe(_update_engine)
  File "C:\Users\Harshad\Documents\pythonProject\venv\lib\site-packages\modin\config\pubsub.py", line 213, in subscribe
    callback(cls)
  File "C:\Users\Harshad\Documents\pythonProject\venv\lib\site-packages\modin\pandas\__init__.py", line 127, in _update_engine
    initialize_ray()
  File "C:\Users\Harshad\Documents\pythonProject\venv\lib\site-packages\modin\core\execution\ray\common\utils.py", line 185, in initialize_ray
    ray.init(**ray_init_kwargs)
  File "C:\Users\Harshad\Documents\pythonProject\venv\lib\site-packages\ray\_private\client_mode_hook.py", line 105, in wrapper
    return func(*args, **kwargs)
  File "C:\Users\Harshad\Documents\pythonProject\venv\lib\site-packages\ray\worker.py", line 922, in init
    ray_params=ray_params)
  File "C:\Users\Harshad\Documents\pythonProject\venv\lib\site-packages\ray\node.py", line 243, in __init__
    "The current node has not been updated within 30 "
Exception: The current node has not been updated within 30 seconds, this could happen because of some of the Ray processes failed to startup.

While I have clearly re-run the code with more time than 30 seconds between them.

When I run it the first time after installing modin and ray, it run fairly well with just the below warnings:

UserWarning: Ray execution environment not yet initialized. Initializing...
To remove this warning, run the following python code before doing dataframe operations:

    import ray
    ray.init()

Then I modified the code to:

import os
os.environ["MODIN_ENGINE"] = "ray"
import modin.pandas as pd
import ray
ray.init()
df = pd.read_csv(r"C:\Users\Harshad\Documents\Files\Data\Pre-processed\data.csv", low_memory=False)

I get this error:

Traceback (most recent call last):
  File "C:\Users\Harshad\Documents\pythonProject\venv\lib\site-packages\ray\node.py", line 240, in __init__
    self.redis_password)
  File "C:\Users\Harshad\Documents\pythonProject\venv\lib\site-packages\ray\_private\services.py", line 328, in wait_for_node
    raise TimeoutError("Timed out while waiting for node to startup.")
TimeoutError: Timed out while waiting for node to startup.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Users/Harshad/Documents/Code/data.py", line 18, in <module>
    low_memory=False)
  File "C:\Users\Harshad\Documents\pythonProject\venv\lib\site-packages\modin\pandas\io.py", line 135, in read_csv
    return _read(**kwargs)
  File "C:\Users\Harshad\Documents\pythonProject\venv\lib\site-packages\modin\pandas\io.py", line 58, in _read
    Engine.subscribe(_update_engine)
  File "C:\Users\Harshad\Documents\pythonProject\venv\lib\site-packages\modin\config\pubsub.py", line 213, in subscribe
    callback(cls)
  File "C:\Users\Harshad\Documents\pythonProject\venv\lib\site-packages\modin\pandas\__init__.py", line 127, in _update_engine
    initialize_ray()
  File "C:\Users\Harshad\Documents\pythonProject\venv\lib\site-packages\modin\core\execution\ray\common\utils.py", line 185, in initialize_ray
    ray.init(**ray_init_kwargs)
  File "C:\Users\Harshad\Documents\pythonProject\venv\lib\site-packages\ray\_private\client_mode_hook.py", line 105, in wrapper
    return func(*args, **kwargs)
  File "C:\Users\Harshad\Documents\pythonProject\venv\lib\site-packages\ray\worker.py", line 922, in init
    ray_params=ray_params)
  File "C:\Users\Harshad\Documents\pythonProject\venv\lib\site-packages\ray\node.py", line 243, in __init__
    "The current node has not been updated within 30 "
Exception: The current node has not been updated within 30 seconds, this could happen because of some of the Ray processes failed to startup

When I looked into the Github for this issue, it turns out to be a bug

How do I resolve these warnings and errors?

Edit: I restarted my pycharm environment which allowed one cycle of rerun. This indicates that its a Pycharm/environment issue?

How can I resolve this issue?

Solution

  • Try initing ray before you import modin:

    import os
os.environ["MODIN_ENGINE"] = "ray"
import ray
ray.init()
import modin.pandas as pd