pythonscikit-learnknnvotingensemble-learning

How to use KNeighborsClassifier in BaggingClassifier & How to solve "KNN doesn't support sample weights issue"

I am new to Sklearn, and I am trying to combine KNN, Decision Tree, SVM, and Gaussian NB for BaggingClassifier.

Part of my code looks like this:

best_KNN = KNeighborsClassifier(n_neighbors=5, p=1)
best_KNN.fit(X_train, y_train)

majority_voting = VotingClassifier(estimators=[('KNN', best_KNN), ('DT', best_DT), ('SVM', best_SVM), ('gaussian', gaussian_NB)], voting='hard')
majority_voting.fit(X_train, y_train)

bagging = BaggingClassifier(base_estimator=majority_voting)
bagging.fit(X_train, y_train)

But this causes an error saying:

TypeError: Underlying estimator KNeighborsClassifier does not support sample weights.

The "bagging" part worked fine if I remove KNN. Does anyone have any idea to solve this issue? Thank you for your time.

Solution

  • In BaggingClassifier you can only use base estimators that support sample weights because it relies on score method, which takes in sample_weightparam.

    You can list all the available classifiers like:

    import inspect 
from sklearn.utils.testing import all_estimators 
for name, clf in all_estimators(type_filter='classifier'): 
    if 'sample_weight' in inspect.getargspec(clf.fit)[0]: 
        print(name)