AttributeError: Can't get attribute 'TunerInternal._validate_overwrite_trainable' on <module 'ray.tune.impl.tuner_internal'

When I am trying to run a Basic Ray Model in our server, Tune Code is breaking. I didnt understand the issue and I didnt find any useful information in internet. Can any one help me with it. FYI, I am able to run them properly when I use local ray cluster which I setup in my local machine.

AttributeError: Can't get attribute 'TunerInternal._validate_overwrite_trainable' on <module 'ray.tune.impl.tuner_internal' from '/home/ray/anaconda3/lib/python3.7/site-packages/ray/tune/impl/tuner_internal.py'>

import ray
from ray.air.config import ScalingConfig
from ray.train.xgboost import XGBoostTrainer

trainer = XGBoostTrainer(
    scaling_config=ScalingConfig(
        # Number of workers to use for data parallelism.
        num_workers=2,
        # Whether to use GPU acceleration.
        use_gpu=False,
    ),
    label_column="y",
    num_boost_round=20,
    params={
        # XGBoost specific params
        "objective": "binary:logistic",
        # "tree_method": "gpu_hist",  # uncomment this to use GPUs.
        "eval_metric": ["logloss", "error"],
    },
    datasets={"train": traindata_ray},
   # preprocessor=preprocessor,
)
result = trainer.fit()
print(result.metrics)

Reference: https://docs.ray.io/en/latest/train/train.html

This is likely due to a ray version mismatch. _validate_overwrite_trainable is available in nightly wheels but not in the latest 2.2.0 release.

Can you verify that you are using the same version of ray across your local machine and cluster?