pythonmachine-learningscikit-learn

How to specify the levels to iterate in a grid search with an ensemble classifier?


I have the following setup but I can't find a way to pass levels to explore in the Grid search for svm* and mlp*:

steps = [('preprocessing', StandardScaler()),
         ('feature_selection', SelectKBest(mutual_info_classif, k=15)),
         ('clf', VotingClassifier(estimators=[("mlp1", mlp1),
                                              ("mlp2", mlp2),
                                              ("mlp3", mlp3),
                                              ("svm1", svm1),
                                              ("svm2", svm2)
                                             ], voting='soft'))
        ]

model = Pipeline(steps=steps)
params = [{
    'preprocessing': [StandardScaler(), MinMaxScaler(), MaxAbsScaler()],
    'feature_selection__score_func': [f_classif, mutual_info_classif]
}]

grid_search = GridSearchCV(model, params, cv=10, scoring='balanced_accuracy', verbose=1, n_jobs=20, refit=True)

Solution

  • Within VotingClassifier, nested parameters can be specified using the following structure:

    clf__<estimator_name>__<hyperparameter_name>
    

    clf refers to the VotingClassifier step in the pipeline. <estimator_name> refers to the name of the estimator in the VotingClassifier (e.g. mlp1, svm1). <hyperparameter_name> refers to the parameter of the individual estimator (e.g. hidden_layer_sizes for MLP or C for SVM).

    Now you can include hyperparameters mlp1, mlp2, mlp3 (for MLP classifiers) and svm1, svm2 (for SVM classifiers) in the params dictionary. e.g.

    params = [{
        # Preprocessing variations
        'preprocessing': [StandardScaler(), MinMaxScaler(), MaxAbsScaler()],
        
        # Feature selection variations
        'feature_selection__score_func': [f_classif, mutual_info_classif],
        
        # Hyperparameters for MLP classifiers
        'clf__mlp1__hidden_layer_sizes': [(50,), (100,), (50, 50)],
        'clf__mlp1__activation': ['relu', 'tanh'],
        'clf__mlp2__hidden_layer_sizes': [(50,), (100,)],
        'clf__mlp2__activation': ['relu', 'tanh'],
        'clf__mlp3__hidden_layer_sizes': [(50, 50), (100, 50)],
        'clf__mlp3__activation': ['relu', 'logistic'],
        
        # Hyperparameters for SVM classifiers
        'clf__svm1__C': [0.01, 0.1, 1, 10],
        'clf__svm1__kernel': ['linear', 'rbf'],
        'clf__svm2__C': [0.1, 1, 10],
        'clf__svm2__kernel': ['rbf', 'sigmoid']
    }]