I've managed to deploy Mistral-7B-v0.1 to a Vertex AI endpoint. For this I've used its model id in the Model Garden ("mistralai/Mistral-7B-v0.1"). But when I use this id for other things, it doesn't work, for example:
from vertexai.language_models import TextGenerationModel
embeddings_model = TextEmbeddingModel.from_pretrained("mistralai/Mistral-7B-v0.1")
text_model = TextGenerationModel.from_pretrained("mistralai/Mistral-7B-v0.1")
Causes an error with the message "Invalid PublisherModel resource name". What id or package should I use to be able to get things such as a TextEmbeddingModel from a model in the Model Garden?
I'm running this from a Vertex AI Workbench instance, so I would expect it to be up-to-date with everything in GCP.
Documentation and examples are very sparse -- most non-GCP sources don't use the Model Garden (yet?) and all the documentation and examples that Google provides itself are all focused on their own models (bison, gecko, etc.)
It seems to me that it's not supported directly. You will have to deploy it then use the endpoint to generate predictions like described in this notebook.
instances = [
{
"prompt": "My favourite condiment is",
"n": 1,
"max_tokens": 200,
"temperature": 1.0,
"top_p": 1.0,
"top_k": 10,
},
]
response = endpoint.predict(instances=instances)
for prediction in response.predictions:
print(prediction)
You can also use the HuggingFace transformers library if you want to use it locally. An example is provided in the same notebook.