I wish to extract and print out a list of llm models from the oobabooga-text-gen-web-ui in python.
Some context first before i head over to my problem.
For those familiar with what ooba is its essentially a Gradio web UI for Large Language Models.
I downloaded and loaded a few llm models onto this web ui. The web ui uses http://127.0.0.1:7860/ to display the web user interface on port 7860.
But if I enable the openai
and api
extensions, and also edit the CMD_FLAGS.txt within the ooba folder into something like:
--listen --api --api-key "enter-your-fake-api-key-here"
the extensions will mimic an Open AI api key by connecting to ooba from a network via port 5000
Here come the problem...
this is my code to view the model list from ooba :
import requests
url = "http://127.0.0.1:5000/v1"
#Model List
headers = {
"Content-Type": "application/json"
}
response = requests.get(f'{url}/internal/model/list',
headers=headers,
verify=False)
print(response.json())
the output should look something like this:
{'model_names': ['L3-8B-Stheno-v3.2', 'L3-8B-Stheno-v3.2_exl2_8h_8bpw', 'L3-8B-Stheno-v3.2_q8_0.gguf', 'MixTAO-7Bx2-MoE-v8.1_q8_0.gguf']}
but instead i got this:
{'detail': 'Unauthorized'}
After some fiddling around I found that if I leave CMD_FLAGS.txt blank the code works as intended and i get the model lists but i dont have access to an API key since its not enable on ooba.
if I do enable it with CMD_FLAGS.txt and typing:
--listen --api --api-key "enter-your-fake-api-key-here"
I'll have access to the openai api key but the code to extract the model list will return as :
{'detail': 'Unauthorized'}
I need the api key enabled from ooba cuz i plan to use the openai.client feature for model interactions. How do I keep the configuration that enables the fake open ai api key but also allows me to extract the model list?
1 day later... Found the answer.
Turns out if i want the api key to be enabled while also being able to view the model list from ooba via the api, I need to add an 'Authorization': f'Bearer {api_key}'
in the header.
The code should look something like this:
import requests
api_key = "enter-your-fake-api-key-here"
url = "http://127.0.0.1:5000/v1"
#Model List
headers = {
"Content-Type": "application/json",
"Authorization": f"Bearer {api_key}"
}
response = requests.get(f'{url}/internal/model/list',
headers=headers,
verify=False)
print(response.json())