I am attempting to use evaluate how Gemini performs on the Learned Hands dataset, which prompts to see if a given post has a specific legal issue included.
My code is the following:
from vertexai.preview import generative_models
from vertexai.preview.generative_models import GenerativeModel
prompt = """
Does the post discuss dealing with domestic violence and abuse, including getting protective orders, enforcing them, understanding abuse, reporting abuse, and getting resources and status if there is abuse?
Post: Like the title states when I was 19 I was an escort for about a year. I was groomed into it by a man I met online. Tony. When I met him I was under the impression he wanted to be my sugar daddy. Instead I was greeted by him and a couple of his girls at a nice hotel. One girl in particular was the star of the show. We were all taken care of by her. Her name was Mo. They promised up to 1,000$ a day for my services. I didn’t have a choice I had nothing. We were forced to see up to 8 guys in a single day. Then cut our profit with Tony. However eventually I came to my senses, took my cash and cut ties. It was incredibly corrupt. They remained bitter at me, sending the occasional threatening message. Very petty nothing worth worrying about. He had my real information such as my email & full name. Cut to about a year later and a former client sends me a news article. Seems Tony got greedy and started his own service. Except he had really fucked up. He was a third striker caught with an underage girl. Mo lured her in so she was also caught and they went to jail. I was completely removed from their life when they made these choices. Cut to three days ago. I get an email from a detective. He knows my full name. He wants to speak to me immediately. Wants to know everything I know. What the heck do I do!!! I’m terrified.
Label:
"""
model = GenerativeModel("gemini-pro")
safety_config = {
generative_models.HarmCategory.HARM_CATEGORY_UNSPECIFIED: generative_models.HarmBlockThreshold.BLOCK_NONE,
generative_models.HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT: generative_models.HarmBlockThreshold.BLOCK_NONE,
generative_models.HarmCategory.HARM_CATEGORY_HARASSMENT: generative_models.HarmBlockThreshold.BLOCK_NONE,
generative_models.HarmCategory.HARM_CATEGORY_HATE_SPEECH: generative_models.HarmBlockThreshold.BLOCK_NONE,
generative_models.HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT: generative_models.HarmBlockThreshold.BLOCK_NONE,
}
chat = model.start_chat()
response = chat.send_message(
prompt,
safety_settings=safety_config,
)
print(response.candidates[0].text)
Unfortunately, this produces the following output:
Traceback (most recent call last):
File "/Users/langston/Documents/Eval/vertex_sample.py", line 25, in <module>
response = chat.send_message(
^^^^^^^^^^^^^^^^^^
File "/Users/langston/miniconda3/envs/prl/lib/python3.11/site-packages/vertexai/generative_models/_generative_models.py", line 709, in send_message
return self._send_message(
^^^^^^^^^^^^^^^^^^^
File "/Users/langston/miniconda3/envs/prl/lib/python3.11/site-packages/vertexai/generative_models/_generative_models.py", line 806, in _send_message
if response.candidates[0].finish_reason not in _SUCCESSFUL_FINISH_REASONS:
~~~~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range
After digging into the raw API response, I assume this error is due to content moderation.
Two questions:
safety_settings?chat.send_message also raises exception when the response is truncated due to exceeding maximum output tokens. On the other hand, model.generate_content does not do any validation and can be used for debugging.