pythonhuggingface-transformerssummarization

'NoneType' object has no attribute 'encode' when loading tokenizer

Error occurs when trying to load Pegasus model for text summarization

from transformers import pipeline, set_seed
pipe = pipeline("summarization", model="google/pegasus-cnn_dailymail")

Error: AttributeError: 'NoneType' object has no attribute 'encode' 

   1588     logger.info("Converting from Tiktoken")
   1589     return TikTokenConverter(
   1590         vocab_file=transformer_tokenizer.vocab_file,
   1591         additional_special_tokens=transformer_tokenizer.additional_special_tokens,
-> 1592     ).converted()
   1593 except Exception:

I've installed tiktoken and restarted the kernel, but the same error occurs

Solution

  • I had same error, and the reason was that I didn't have sentencepiece package installed.

    So

    pip install sentencepiece

    solved the problem