weaviatevector-database

Weaviate Gives Rate Limit Error while adding objects


I am using free trial of weaviate for testing purpose. I have nearly 300 objects, which I wanted to add them to weaviate database. I am using hugging face api for vectorizing. The hugging face API works fine. But, After inserting 15-20 objects, weaviate gives rate limit error as:

{‘error’: [{‘message’: ‘update vector: failed with status: 429 error: Rate limit reached. Please log in or use your apiToken’}]}

I have added throttling to it, which inserts 1 object to weaviate every 10 seconds. But, then to, after adding 15-20 objects, weaviate gives above error.

Can anyone please tell me, how much data or request does weaviate accept. Or like, after how many request it gives rate limit error. Or what is the condition for rate limit error.

I tried finding it in documentation, but I just understood , we need to use throttling , which I am using.


Solution

  • The rate limit comes from HuggingFace.

    I believe, if you don't provide an apiToken key (as mentioned in the error message), you are limited to 10 vectorisations per minute.

    You can either:

    1. run in a loop, and sleep for ~60s every 10 inserts
    2. get an API key from HuggingFace, and provide in the header, as: "X-Huggingface-Api-Key". For example, if you are using a Python client, this would look like this:
    client = weaviate.connect_to_local(
        headers={
            "X-HuggingFace-Api-Key": "YOUR_HUGGINGFACE_APIKEY",
        }
    )