I adapted this code from https://www.datacamp.com/tutorial/llama-cpp-tutorial
from llama_cpp import Llama
# GLOBAL VARIABLES
my_model_path = "./model/zephyr-7b-beta.Q4_0.gguf"
CONTEXT_SIZE = 512
# LOAD THE MODEL
zephyr_model = Llama(model_path=my_model_path,
n_ctx=CONTEXT_SIZE)
def generate_text_from_prompt(user_prompt,
max_tokens = 100,
temperature = 0.3,
top_p = 0.1,
echo = True,
stop = ["Q", "\n"]):
# Define the parameters
model_output = zephyr_model(
user_prompt,
max_tokens=max_tokens,
temperature=temperature,
top_p=top_p,
echo=echo,
stop=stop,
)
return model_output
if __name__ == "__main__":
my_prompt = "What do you think about fishing for catfish?"
zephyr_model_response = generate_text_from_prompt(my_prompt)
final_result = zephyr_model_response["choices"][0]["text"].strip()
print(final_result)
How do I print the response to the prompt? It prints a huge amount of information but I don't see the response.
When I print model_output I get
{'id': 'cmpl-f13d4d89-5f25-4c01-b274-97df4ebaba84', 'object': 'text_completion', 'created': 1715958626, 'model': './model/zephyr-7b-beta.Q4_0.gguf', 'choices': [{'text': 'What do you think about fishing for catfish?', 'index': 0, 'logprobs': None, 'finish_reason': 'stop'}], 'usage': {'prompt_tokens': 11, 'completion_tokens': 1, 'total_tokens': 12}}
The answer is... the code is correct. Sometimes the model simply doesn't generate an answer, due to the nature of the model and the arguments provided with the prompt.
If I change the top_p and temperature arguments to be the the most forgiving and random:
def generate_text_from_prompt(user_prompt,
max_tokens = None,
temperature = 1,
top_p = 1,
echo = True,
stop = ["Q", "\n"]):
and I prompt the model 50 times:
for i in range(0, 50):
my_prompt = "What is a bass?"
zephyr_model_response = generate_text_from_prompt(my_prompt)
final_result = zephyr_model_response["choices"][0]["text"].strip()
print(i, ":", final_result)
Sometimes we get answers and sometimes we don't:
0 : What is a bass?
1 : What is a bass?
2 : What is a bass?
3 : What is a bass?
4 : What is a bass?
5 : What is a bass? It’s simple: A bass is the lowest sounding instrument in an orchestra, usually tuned to an E. However, a lot of people aren’t quite sure how to classify the double bass, and where exactly it should fall in the musical world.
6 : What is a bass?
7 : What is a bass?
8 : What is a bass?
9 : What is a bass? The word "bass" derives from the Greek 'basso' meaning deep or low. But what defines a bass as a bass?
10 : What is a bass?
11 : What is a bass? The word itself conjures up images of big, lumpish beasts with long-slung guitars and deep voices. But while the likes of Steve Harris and Glenn Hughes have fitted that description down the years, what about those bass players who don't necessarily fit into that mould?
12 : What is a bass?
13 : What is a bass?
14 : What is a bass?
15 : What is a bass? The common carp (Cyprinus carpio) is the species most usually meant by carpers and anglers talking or composing about “bass”. However, it isn’t a member of the bass family (Moronidae). Carps are members of the minnow family (Cyprinidae) along with many other kinds.
16 : What is a bass?