I understand from the documentation that my inputs should have the following format:
{
"request":{
"contents": [
{"role": "user", "parts": [
{"text": "What is the relation between the following video and image samples?"}
{"fileData": {"fileUri": "gs://cloud-samples-data/generative-ai/video/animals.mp4", "mimeType": "video/mp4"}},
{"fileData": {"fileUri": "gs://cloud-samples-data/generative-ai/image/cricket.jpeg", "mimeType": "image/jpeg"}}]}]}}
Since I'm only processing text, I'm ignoring the fileData
parts. However, for my use case, I need to provide system instructions and few shot examples.
I have checked the documentation for hours but couldn't find any useful info.
I have tried the following data structure, but it didn't work as the system
role is not available.
{
"request":{
"contents": [
{"role": "system", "parts": [{"text": "System prompt"}]},
{"role": "user", "parts": [{"text": "Example input"}]},
{"role": "assistant", "parts": [{"text": "Example output"}]},
{"role": "user", "parts": [{"text": "The actual query"}]},
]}}
Next thing I tried was to add the system prompt to the first example, but looking at the output if looks like my system prompt is being ignored.
{
"request":{
"contents": [
{"role": "user", "parts": [
{"text": "System prompt"},
{"text": "Example input"}]},
{"role": "assistant", "parts": [{"text": "Example output"}]},
{"role": "user", "parts": [{"text": "The actual query"}]},
]}}
I'm not 100% sure the approach above is wrong but I'd rather not guess and have some clear guidance on how to proceed.
So my question is, does anyone know the right way to run batch predictions with few shot examples in Vertex AI.
FYI, the model I'm using for inference is gemini-1.5-flash-002
.
To run batch predictions with few-shot examples in vertex AI. Specify your BigQuery input table, model, and output location. The batch prediction job and your table must be in the same region.
Here an example of input data (JSON):
{
"contents": [
{
"role": "user",
"parts": [
{
"text": "Give me a recipe for banana bread."
}
]
}
],
"system_instruction": {
"parts": [
{
"text": "You are a chef."
}
]
}
}
Instead of having a role: “system” , we have a "system_instruction" variable where we can put the system prompt we wanted.
Additionally here are some examples of using batch prediction and system instructions.