pythonmachine-learningxgboostshap

Calculation of expected_value in SHAP explanations of XGBoost Classifier


How do we make sense of SHAP explainer.expected_value? Why is it not the same with y_train.mean() after sigmoid transformation?

Below is a summary of the code for quick reference. Full code available in this notebook: https://github.com/MenaWANG/ML_toy_examples/blob/main/explain%20models/shap_XGB_classification.ipynb

model = xgb.XGBClassifier()
model.fit(X_train, y_train)
explainer = shap.Explainer(model)
shap_test = explainer(X_test)
shap_df = pd.DataFrame(shap_test.values)

#For each case, if we add up shap values across all features plus the expected value, we can get the margin for that case, which then can be transformed to return the predicted prob for that case:
np.isclose(model.predict(X_test, output_margin=True),explainer.expected_value + shap_df.sum(axis=1))
#True

But why isn't the below true? Why after sigmoid transformation, the explainer.expected_value is not the same with y_train.mean() for XGBoost classifiers?

expit(explainer.expected_value) == y_train.mean()
#False

Solution

  • SHAP is guaranteed to be additive in raw space (logits). To understand why additivity in raw scores doesn't extend to additivity in class predictions you may think for a while why exp(x+y) != exp(x) + exp(y)

    Re: Just keen to understand how was explainer.expected_value calculated for XGBoost classifier. Do you happen to know?

    As I stated in comments expected value comes either from the model trees or from your data.

    Let's try reproducible:

    from sklearn.model_selection import train_test_split
    import xgboost
    import shap
    
    X, y = shap.datasets.adult()
    X_display, y_display = shap.datasets.adult(display=True)
    
    # create a train/test split
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=7)
    d_train = xgboost.DMatrix(X_train, label=y_train)
    d_test = xgboost.DMatrix(X_test, label=y_test)
    
    params = {
        "eta": 0.01,
        "objective": "binary:logistic",
        "subsample": 0.5,
        "base_score": np.mean(y_train),
        "eval_metric": "logloss",
    }
    model = xgboost.train(
        params,
        d_train,
        num_boost_round=5000,
        evals=[(d_test, "test")],
        verbose_eval=100,
        early_stopping_rounds=20,
    )
    

    Case 1. No data available, trees only.

    explainer = shap.TreeExplainer(model)
    ev_trees = explainer.expected_value[0]
    
    from shap.explainers._tree import XGBTreeModelLoader
    
    xgb_loader = XGBTreeModelLoader(model)
    ts = xgb_loader.get_trees()
    
    v = []
    for t in ts:
        v.append(t.values[0][0])
    sv = sum(v)
    
    import struct
    from scipy.special import logit
    size = struct.calcsize('f')
    buffer = model.save_raw().lstrip(b'binf')
    v = struct.unpack('f', buffer[0:0+size])[0]
    # if objective "binary:logistic" or "reg:logistic" 
    bv = logit(v)
    
    ev_trees_raw = sv+bv
    
    np.isclose(ev_trees, ev_trees_raw)
    

    True
    

    Case 2. Background data set supplied.

    background = X_train[:100]
    
    explainer = shap.TreeExplainer(model, background)
    ev_background = explainer.expected_value
    

    Take a note that:

    np.isclose(ev_trees, ev_background)
    

    False
    

    but

    d_train_background = xgboost.DMatrix(background, y_train[:100])
    preds = model.predict(d_train_background, pred_contribs = True)
    
    np.isclose(ev_background, preds.sum(1).mean())
    

    True
    

    or simply

    output_margin = model.predict(d_train_background, output_margin=True)
    np.isclose(ev_background, output_margin.mean())
    

    True