I am applying a Decision Tree to a data set, using sklearn.
In sklearn there is a parameter that sets the depth of the tree:
dtree = DecisionTreeClassifier(max_depth=10).
My question is: How does the max_depth parameter influence the model? How does a high/low max_depth help in predicting the test data more accurately?
max_depth is what the name suggests: The maximum depth that you allow the tree to grow to. The deeper you allow, the more complex your model will become.
For training error, it is easy to see what will happen. If you increase max_depth, training error will always go down (or at least not go up).
For testing error, it gets less obvious. If you set max_depth too high, then the decision tree might simply overfit the training data without capturing useful patterns as we would like; this will cause testing error to increase. But if you set it too low, that is not good as well; then you might be giving the decision tree too little flexibility to capture the patterns and interactions in the training data. This will also cause the testing error to increase.
There is a nice golden spot in between the extremes of too-high and too-low. Usually, the modeller would consider the max_depth as a hyper-parameter, and use some sort of grid/random search with cross-validation to find a good number for max_depth.