pythonnamed-entity-recognitionprecision-recallspacy-3

How to generate Precision, Recall and F-score in Named Entity Recognition using Spacy v3? Seeking ents_p, ents_r, ents_f for a small custom NER model


The example code is given below, you may add one or more entities in this example for training purposes (You may also use a blank model with small examples for demonstration). I am seeking a complete working solution for custom NER model evaluation (precision, recall, f-score), Thanks in advance to all NLP experts.

import spacy
nlp = spacy.load('en_core_web_sm')
example_text = "Agra is famous for Tajmahal, The CEO of Facebook will visit India shortly to meet Murari Mahaseth and to visit Tajmahal"
ner = nlp.get_pipe("ner")
doc = nlp(example_text)
for e in doc.ents:
    print(e.text + ' - ' + str(e.start_char) + ' - ' + str(e.end_char) + ' - ' + e.label_ + ' - ' + str(
        spacy.explain(e.label_)))

Solution

  • I will give a brief example :

    import spacy
    from spacy.scorer import Scorer
    from spacy.tokens import Doc
    from spacy.training.example import Example
    
    examples = [
        ('Who is Talha Tayyab?',
         {(7, 19, 'PERSON')}),
        ('I like London and Berlin.',
         {(7, 13, 'LOC'), (18, 24, 'LOC')}),
         ('Agra is famous for Tajmahal, The CEO of Facebook will visit India shortly to meet Murari Mahaseth and to visit Tajmahal.',
         {(0, 4, 'LOC'), (40, 48, 'ORG'), (60, 65, 'GPE'), (82, 97, 'PERSON'), (111, 119, 'GPE')})
    ]
    
    def my_evaluate(ner_model, examples):
        scorer = Scorer()
        example = []
        for input_, annotations in examples:
            pred = ner_model(input_)
            print(pred,annotations)
            temp = Example.from_dict(pred, dict.fromkeys(annotations))
            example.append(temp)
        scores = scorer.score(example)
        return scores
    
    ner_model = spacy.load('en_core_web_sm') # for spaCy's pretrained use 'en_core_web_sm'
    results = my_evaluate(ner_model, examples)
    print(results)
    

    As, I said this is just an example you can make changes according to your needs.