pythonnlpbert-language-modelelmo

Iterating through multiple files with BERT for QA returns nothing


I am trying to ease my job. I need to do some analysis on the answers BERT gives me for thousands of files. My main objective is to iterate through every file and ask A question.

I have been trying to automate it with the following code

import os

directory = '/content/dva/'

for filename in os.listdir(directory):
with open(directory + filename) as infile:
    try:

      nlp({
    'question': 'How is artificial intelligence being used in real time health delivery?',
    'context': data
})
    except:
        print(filename + ' is throwing an error')

The above code returns nothing. Yet, if I do them one by one. It works fine. So I tried changing it.

x = ["How is artificial intelligence being used in real time health delivery?",\
     "What adjunctive or supportive methods can help patients?",\
     "How does hypertension affect patients?",\
      "What does the computer do?"]

y = [item.strip() for item in x]

def testing(theList):
  nlp = pipeline('question-answering')

  for each_element in theList:
    nlp({'question': each_element,'context': data})


  

testing(y) # returns nothing
print(testing(y)) # returns None

Does anyone have any insights? The above code works perfectly for Allen's ELMo.


Solution

  • For some reason, when looping through all files, print() actually does return the answer. It is weird, because usually you do not need to call print to make it work.

    Working code:

    import os
    
    directory = '/content/dva/'
    
    for filename in os.listdir(directory):
    with open(directory + filename) as infile:
    try:
    
      print(nlp({
    'question': 'How is artificial intelligence being used in real time health delivery?',
    'context': data
    }))
        except:
            print(filename + ' is throwing an error')