What is the difference between mteval-v13a.pl and NLTK BLEU?
There is an implementation of BLEU score in Python NLTK,
nltk.translate.bleu_score.corpus_bleu
But I am not sure if it is the same as the mtevalv13a.pl script.
What is the difference between them?
TL;DR
Use https://github.com/mjpost/sacrebleu when evaluating Machine Translation systems.
In Short
No, the BLEU in NLTK isn't the exactly the same as the mteval-13a.perl.
But it can get really close, see https://github.com/nltk/nltk/issues/1330#issuecomment-256237324
nltk.translate.corpus_bleucorresponds tomteval-13a.plup to the 4th order of ngram with some floating point discrepancies
The details of the comparison and the dataset used can be downloaded from https://github.com/nltk/nltk_data/blob/gh-pages/packages/models/wmt15_eval.zip or:
import nltk
nltk.download('wmt15_eval')
The major differences:
In Long
There are several difference between mteval-13a.pl and nltk.translate.corpus_bleu:
The first difference is the fact that
mteval-13a.plcomes with its own NIST tokenizer while the NLTK version of BLEU is the implementation of the metric and assumes that input is pre-tokenized.- BTW, this ongoing PR will bridge the gap between NLTK and NIST tokenizers
The other major difference is that
mteval-13a.plexpects the input to be in.sgmformat while NLTK BLEU takes in python list of lists of strings, see the README.txt in the zipball here for more information of how to convert textfile to SGM.mteval-13a.plexpects an ngram order of at least 1-4. If the minimum ngram order for the sentence/corpus is less than 4, it will return a 0 probability which is amath.log(float('-inf')). To emulate this behavior, NLTK has a put an_emulate_multibleuflag:mteval-13a.plis able to generate NIST scores while NLTK doesn't have NIST score implementation (at least not yet)- NIST score in NLTK is upcoming in this PR
Other than the differences, NLTK BLEU scores packed in more features:
to handle fringe cases that the original BLEU (Papineni, 2002) overlooked
Also to handle fringe cases where the largest order of Ngram is < 4, the uniform weights of the individual ngram precision will be reweighted such that the mass of the weights sums to 1.0
while NIST has a smoothing method for geometric sequence smoothing, NLTK has an equivalent object with the same smoothing method and even more smoothing methods to handle sentence level BLEU from Chen and Collin, 2014
Lastly to validate the features added in NLTK's version of BLEU, a regression test is added to accounts for them, see https://github.com/nltk/nltk/blob/develop/nltk/test/unit/translate/test_bleu.py
- Calculating the partial derivative of PyTorch model output with respect to neurons pre-activation
- Gradient Descent blowing up in linear regression
- What does model.train() do in PyTorch?
- How to use kmeans in python with Pearson correlation as distance measure, similar to MATLAB?
- sklearn StandardScaler, doesn't allow direct transform, we need to fit_transform
- Retrieve final hidden activation layer output from sklearn's MLPClassifier
- ValueError when fitting model: Expected 2D array, got 1D array instead
- How to convert yolo annotations to coco format. Json?
- Neural Net: Loss decreasing, but accuracy stays exactly the same
- Increasing The Amount Of Neurons In Hidden Layer
- PyTorch Linear MNIST model training error
- How do I format my a tensorflow dataset for a multi output model?
- RuntimeError: DataLoader worker exited unexpectedly
- Macro VS Micro VS Weighted VS Samples F1 Score
- scikit-learn: Predicting new points with DBSCAN
- How to save learned weights/parameters of PCA and T-SNE in Sklearn
- keras 2.4. producing completely different output than 2.3.1
- Pyannote: Load and Apply Speaker Diarization Offline
- Shap force plot not displaying figure: shap.plots._force.AdditiveForceVisualizer
- K-Means Clustering Output seems wrong, how can it be explained?
- Augment MNIST dataset tensorflow
- What are the pros and cons between get_dummies (Pandas) and OneHotEncoder (Scikit-learn)?
- What parameter do I need to change for it to match requirements?
- Optuna sample fixed parameter depending on another parameter
- TF Lite shows only zeros on prediction
- How to use the OneVsRestClassifier in Scikit-learn to analyse the performance of predicting each individual class with multi-class classification?
- What happens if you use LinearRegression with OneVsRestClassifier
- Faster Kmeans Clustering on High-dimensional Data with GPU Support
- NLTK classifier for integer features?
- KeyError in Decision Tree during prediction