String comparison with BERT seems to ignore "not" in sentence

I implemented a string comparison method using SentenceTransformers and BERT like following

from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity

model = SentenceTransformer('sentence-transformers/all-distilroberta-v1')

sentences = [
    "I'm a good person",
    "I'm not a good person"
]

sentence_embeddings = model.encode(sentences)

cosine_similarity(
    [sentence_embeddings[0]],
    sentence_embeddings[1:]
)

Notice how my sentence examples are very similar but with the opposite meaning. The problem is the cosine similarity returns 0.9, indicating that these two strings are very similar in context when I expected it to return something closer to zero, as they have the opposite meanings.

How can I adapt my code to return a more accurate result?

TL;DR: NLI is all you need

First, the cosine similarity is reasonably high, because the sentences are similar in the following sense:

• They are about the same topic (evaluation of a person)
• They are about the same subject ("I") and the same property ("being a good person")
• They have similar syntactic structure
• They have almost the same vocabulary

So, from the formal point of view, they should be considered similar. Moreover, from the practical point of view, they should often be considered similar. For example, if you google "GMO are causing cancer", you might find that the text with label "GMO are not causing cancer" is relevant.

Second, if you want to measure logical connection between sentences, cosine similarity of embeddings is just not expressive enough. This is because embeddings contain lots of semantic stylistic, lexical and syntactic information, but they are fixed-size (768-dimensional, in your case), so they cannot contain complete information about the meaning of both sentences. So you need another model with the following properties:

1. It encodes both texts simultaneously, so it compares the texts themselves, not just their fixed-size embeddings
2. It is explicitly trained to evaluate logical connection between sentences

The task of assesing logical connection between texts is called natural language inference (NLI), and its most common formulation is recognizing textual entailment (RTE): it is the problem of predicting whether the first sentence entails the second one.

There are lots of models trained for this task in the Huggingface repo, with roberta-large-mnli being a good one. You can use it to evaluate equivalence of two texts. If each text entails another, they are equivalent, so you can estimate the degree of equivalence as the product of the entailment scores in both directions.

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained("roberta-large-mnli")
model = AutoModelForSequenceClassification.from_pretrained("roberta-large-mnli")

def test_entailment(text1, text2):
    batch = tokenizer(text1, text2, return_tensors='pt').to(model.device)
    with torch.no_grad():
        proba = torch.softmax(model(**batch).logits, -1)
    return proba.cpu().numpy()[0, model.config.label2id['ENTAILMENT']]

def test_equivalence(text1, text2):
    return test_entailment(text1, text2) * test_entailment(text2, text1)

print(test_equivalence("I'm a good person", "I'm not a good person"))  # 2.0751484e-07
print(test_equivalence("I'm a good person", "You are a good person"))  # 0.49342492
print(test_equivalence("I'm a good person", "I'm not a bad person"))   # 0.94236994