I am building a binary text classifier with NLTK, using its movie_review corpus that has 1000 positive and 1000 negative review. It's working on the test set and the metrics (TPR, TNR) are fine, but when I manually input a review, it always with 99%+ certainty thinks it's negative, even if I put only a single word, that is considered by the model a positive feature.
import nltk
import random
import collections
from nltk.corpus import movie_reviews
documents = []
for category in movie_reviews.categories():
for fileid in movie_reviews.fileids(category):
documents.append((movie_reviews.words(fileid), category))
random.seed(432)
random.shuffle(documents)
all_words = [word.lower() for word in movie_reviews.words()]
from nltk.corpus import stopwords
stopwords_english = stopwords.words('english')
all_words_clean = []
for word in all_words:
if word not in stopwords_english and word.isalpha():
all_words_clean.append(word)
all_words_frequency = nltk.FreqDist(all_words_clean)
most_common_words = all_words_frequency.most_common(2000)
word_features = [item[0] for item in most_common_words]
def document_features(document):
document_words = set(document)
features = {}
for word in word_features:
features['contains(%s)' % word] = (word in document_words)
return features
feature_set = [(document_features(doc), category) for (doc, category) in documents]
test_set = feature_set[:400]
train_set = feature_set[400:]
from nltk import NaiveBayesClassifier
classifier = NaiveBayesClassifier.train(train_set)
print(classifier.show_most_informative_features(10))
from nltk import classify
accuracy = classify.accuracy(classifier, test_set)
print('Accuracy:', accuracy)
refsets = collections.defaultdict(set)
testsets = collections.defaultdict(set)
for i, (features, label) in enumerate(test_set):
refsets[label].add(i)
observed = classifier.classify(features)
testsets[observed].add(i)
print('Precision:', nltk.precision(refsets['pos'], testsets['pos']))
print('Recall:', nltk.recall(refsets['pos'], testsets['pos']))
print('F1:', nltk.f_measure(refsets['pos'], testsets['pos']))
print('TNR:', nltk.precision(refsets['neg'], testsets['neg']))
print('NPV:', nltk.recall(refsets['neg'], testsets['neg']))
print('bACC:', (nltk.recall(refsets['pos'], testsets['pos'])+
nltk.precision(refsets['neg'], testsets['neg']))/2)
print('nF1:', nltk.f_measure(refsets['neg'], testsets['neg']))
from nltk.tokenize import word_tokenize
def class_new(new_text):
new_feat = document_features(word_tokenize(new_text))
prob_result = classifier.prob_classify(new_feat)
print (prob_result.max())
print (prob_result.prob("neg"))
print (prob_result.prob("pos"))
The output is:
Most Informative Features
contains(seagal) = True neg : pos = 11.5 : 1.0
contains(outstanding) = True pos : neg = 9.5 : 1.0
contains(damon) = True pos : neg = 8.2 : 1.0
contains(lame) = True neg : pos = 6.1 : 1.0
contains(wonderfully) = True pos : neg = 6.0 : 1.0
contains(poorly) = True neg : pos = 6.0 : 1.0
contains(wasted) = True neg : pos = 5.6 : 1.0
contains(flynt) = True pos : neg = 5.4 : 1.0
contains(ridiculous) = True neg : pos = 5.4 : 1.0
contains(unfunny) = True neg : pos = 5.0 : 1.0
None
Accuracy: 0.8025
Precision: 0.7988505747126436
Recall: 0.7595628415300546
F1: 0.7787114845938375
TNR: 0.8053097345132744
NPV: 0.8387096774193549
bACC: 0.7824362880216644
nF1: 0.8216704288939051
And here's the input:
class_new('outstanding')
neg
0.9999795728097999
2.0427190192409225e-05
I'm following this link and the author also gets the same thing. Am I doing something wrong? Is this because of the feature set? If so, why are the metrics showing that the model works fine?
TL;DR: The way you try to test your algorithm does not reflect the way it was trained. When you enter a text similar to a review that your nltk dataset provides, you get positive predictions.
Your Naive Bayes classifier not only calculates its probabilities based on words which appear in the input text, but also words which do not appear:
print(classifier.show_most_informative_features(2000))
…
contains(film) = False neg : pos = 1.3 : 1.0
contains(best) = False neg : pos = 1.3 : 1.0
contains(many) = False neg : pos = 1.3 : 1.0
contains(well) = False neg : pos = 1.2 : 1.0
…
There are about 100 of these conditions; thus, when you enter one word in your classifier you stack that one word against 100 missing words. Given the results you get, these "negative occurances" predominantly assign a given text to the "negative" category. Note that your algorithm has been trained on documents which contain, on average, 800 words. Therefore, to get useful predictions out of your classifier, you also need to provide a document which is similar to the data it was trained on.
# count the average number of words in a document
documents_wordcount = []
for category in movie_reviews.categories():
for fileid in movie_reviews.fileids(category):
documents_wordcount.append(len([word for word in movie_reviews.words(fileid)]))
print(sum(documents_wordcount)/len(documents_wordcount)) # 791.91