pythonhuggingfacehuggingface-datasets

How do I convert a list of dictionaries to a Huggingface Dataset object?

I have a list of dictionaries:

print(type(train_dataset))
>>> <class 'list'>

print(len(train_dataset))
>>> 4000

train_dataset[0]
>>>
{'id': '7',
 'question': {'stem': 'Who is A',
  'choices': [{'text': 'A is X', 'label': 'A'},
   {'text': 'A is not B', 'label': 'D'}]},
 'answerKey': 'D'}

How can I convert this to a huggingface Dataset object? From their website it seems like you can only convert pandas df (dataset = Dataset.from_pandas(df)) or a dictionary ( dataset = Dataset.from_dict(my_dict)), but it's not clear how to use a list of dictionaries

Solution

  • From here: https://discuss.huggingface.co/t/convert-a-list-of-dictionaries-to-hugging-face-dataset-object/14670

    datasets.Dataset.from_pandas(pd.DataFrame(data=your_data))