I am performing hyperparameter tuning on a KNN Classifier without feature selection. However, I want to compare the performance with feature selection and without feature selection. My data has been split in training, validation and testing set.
In general, do I also have to use the same hyperparameter value when performing feature selection or do I have to perform another hyperparameter tuning in the dataset with feature selection in order to compare performance for this problem? I do not want to use cross-validation but the training, validation and testing sets.
To make a proper comparison you should only change one element between the two groups you are comparing. So if you tune hyperparameters in one group without feature selection, then you should be using the same hyperparameters in the second group with feature selection.
This way you are only looking at the effects of feature selection.