spacytraining-dataevaluationtest-data

Spacy train dev and test data


Does spaCy use dev-data to tune hyper-parameters? Or dev-data is totally out of the training process, and so equivalent to test data? Following the standard greatly explained here, validation data and test data are different. Please is someone can clarify which is the case for spaCy under the referred standard. Thanks a lot.


Solution

  • The spacy core library does not do any hyperparameter tuning. For spacy train, the dev data is used for the evaluation displayed during training, to select the best model, and for early stopping (the early stopping setting is called patience).