pythoncatboostshap

Shap library for model explanation gives values outside range of 0 to 1


I have a catboost model and I am trying to draw a decision plot for some of the sample records to see which features are driving the prediction towards 0 or 1. However, I noticed that the range of the model output value is not 0-1 but rather it ranges from negative 2 to positive 2

Here is a screenshot of 1 such example where the prediction probability is 0.82 but the model output value shows 1.5 ( I have redacted my feature names because of privacy reasons)

enter image description here

Why does the range not match with the probability range of 0-1?

Here is my code snippet -

explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(transformer_pipeline)
# generate decision plot for 1st record as an example
shap.decision_plot(explainer.expected_value, shap_values[0,:], transformer_pipeline.iloc[0,:])

Solution

  • By default, the TreeExplainer does not explain the predicted probabilities, but rather the raw output of the trees, which varies by model. This should explain why you don't recover the probability value of 0.82 for this specific sample.

    To work with probabilities, you have to change in shap.TreeExplainer() the value of model_output into either "predict_proba", which will explain the output obtained by calling model.predict_proba(), or "probability", which explains the output of the model transformed into probability space. See the function documentation for more details.

    Notice that things seem to be a bit more complicated for multiclass classification cases, see the discussion at https://github.com/slundberg/shap/issues/756