huggingface-transformers

Convert tokenizer.json to tokenizer.model

I need to convert a hugginface repo https://huggingface.co/LeoLM/leo-hessianai-13b to llama.cpp gguf format. Part of the process needs a tokenizer.model file (which funnily enough is present in https://huggingface.co/LeoLM/leo-hessianai-7b). It does include a tokenizer.json file though which is the same just another format (hugginface format).

So how can I convert a tokenizer.json to a tokenizer.model file?

Solution

  • Nevermind they just added it an hour ago