google-cloud-platformgoogle-cloud-vertex-aimistral-7b

How to use Vertex AI Model Garden models in Workbench?


I've managed to deploy Mistral-7B-v0.1 to a Vertex AI endpoint. For this I've used its model id in the Model Garden ("mistralai/Mistral-7B-v0.1"). But when I use this id for other things, it doesn't work, for example:

from vertexai.language_models import TextGenerationModel

embeddings_model = TextEmbeddingModel.from_pretrained("mistralai/Mistral-7B-v0.1")
text_model = TextGenerationModel.from_pretrained("mistralai/Mistral-7B-v0.1")

Causes an error with the message "Invalid PublisherModel resource name". What id or package should I use to be able to get things such as a TextEmbeddingModel from a model in the Model Garden?

I'm running this from a Vertex AI Workbench instance, so I would expect it to be up-to-date with everything in GCP.

Documentation and examples are very sparse -- most non-GCP sources don't use the Model Garden (yet?) and all the documentation and examples that Google provides itself are all focused on their own models (bison, gecko, etc.)


Solution

  • It seems to me that it's not supported directly. You will have to deploy it then use the endpoint to generate predictions like described in this notebook.

    instances = [
        {
            "prompt": "My favourite condiment is",
            "n": 1,
            "max_tokens": 200,
            "temperature": 1.0,
            "top_p": 1.0,
            "top_k": 10,
        },
    ]
    response = endpoint.predict(instances=instances)
    
    for prediction in response.predictions:
        print(prediction)
    

    You can also use the HuggingFace transformers library if you want to use it locally. An example is provided in the same notebook.