How to get probability of an answer using BERT model and is there a way to ask multiple questions for a context

I am new to AI models and currently experimenting with the QandA model. Particularly I am interested in following 2 models. 1. from transformers import BertForQuestionAnswering
2. from simpletransformers.question_answering import QuestionAnsweringModel

Using option 1 BertForQuestionAnswering I am getting the desired results. However I can ask only one question at a time. Also I am not getting the probability of the answer.

below is the code for BertForQuestionAnswering from transformers.

from transformers import BertTokenizer, BertForQuestionAnswering
import torch

tokenizer = BertTokenizer.from_pretrained('bert-large-uncased-whole-word-masking-finetuned-squad')
model = BertForQuestionAnswering.from_pretrained('bert-large-uncased-whole-word-masking-finetuned-squad')


input_ids = tokenizer.encode('Sky color?', 'Today the sky is blue. and it is cold out there')
tokens = tokenizer.convert_ids_to_tokens(input_ids)
sep_index = input_ids.index(tokenizer.sep_token_id)
num_seg_a = sep_index + 1
num_seg_b = len(input_ids) - num_seg_a
segment_ids = [0]*num_seg_a + [1]*num_seg_b
assert len(segment_ids) == len(input_ids)
outputs = model(torch.tensor([input_ids]), 
                            token_type_ids=torch.tensor([segment_ids]), 
                            return_dict=True)
start_scores = outputs.start_logits
end_scores = outputs.end_logits
answer_start = torch.argmax(start_scores)
answer_end = torch.argmax(end_scores)
answer = ' '.join(tokens[answer_start:answer_end+1])
print(answer)

Here is the output: blue

Where as using option 2 QuestionAnsweringModel from simpletransformers, I can put multiple questions at a time and also getting probability of the answer.

below is the code for QuestionAnsweringModel from simpletransformers

from simpletransformers.question_answering import QuestionAnsweringModel
model = QuestionAnsweringModel('distilbert', 'distilbert-base-uncased-distilled-squad', use_cuda=False)

question_data = {
        'qas': [{
            'question': 'Sky color?',
            'id': 0,
        },
        {
            'question': 'weather?',
            'id': 1,
        }
        ],
        'context': 'Today the sky is blue. and it is cold out there'
    }

prediction = model.predict([question_data])
output = {'result': list(prediction)}
print(output)

Here is the output:

{
   "result":[
      [
         {
            "id":0,
            "answer":["blue", "the sky is blue", "blue."]
         },
         {
            "id":1,
            "answer":["cold", "it is cold", "cold out there"]
         }
      ],
      [
         {
            "id":0,
            "probability":[0.8834650211919095,0.0653234009794176,0.031404456093241565]
         },
         {
            "id":1,
            "probability":[0.6851319220199236,0.18145769901523698,0.05004994980074798]
         }
      ]
   ]
}

As you can see, For the same context I can ask multiple questions at a time and get the probability for each answer.

Is there a way I can get similar output for the BERT model in option#1. I need a way to set multiple questions to a context and also need probability for each answer in the response.

Any help would be greatly appreciated.

You can use the huggingface question answering pipeline to achieve that:

from transformers import pipeline

model_checkpoint = 'bert-large-uncased-whole-word-masking-finetuned-squad' qa_pipeline =  pipeline('question-answering', model=model_checkpoint, tokenizer=model_checkpoint)

qa_pipeline(question=['Sky color?', 'weather?'], context='Today the sky is blue. and it is cold out there')

Output:

[{'score': 0.9102755188941956, 'start': 17, 'end': 21, 'answer': 'blue'},
 {'score': 0.49744659662246704, 'start': 33, 'end': 37, 'answer': 'cold'}]