Fetch Data from Azure Blob Storage to Azure AI Search - RAG

I am developing a RAG-based chatbot to search through my content. I have uploaded all the contents to blob storage and imported the blob storage into the Azure AI Search service. I attempted to use an index/indexer, but the index operates based on fields only, whereas I want to search through everything. Is there an alternative way to achieve this?

I also want any new data added to my blob container to be automatically included in Azure AI Search.

I attempted to use an index/indexer, but the index operates based on fields only, whereas I want to search through everything. Is there an alternative way to achieve this?

Based on your capture, where you only have 1 index field called 'id', I guess you incorrectly understood the way an index is used. You need to define your index structure with the appropriate fields first, given what you want to achieve:

• if you want to index documents to search them later, you need to have your documents' content in one of your index fields (probably using a string type), and set this field as 'searchable'
• if you also want to be able to search using vector (or hybrid mode, combining keyword + vector), you will also need a different column containing those vectors
• if you want to easily get your document title, add another field
• if you want to easily get your document name / filename, add another field
• etc etc

And once you have listed what you need from your files, and defined and created your index structure, then you will use an 'indexer' to fill your index with your data.

For example, when you use the "Add your data" tab in Azure OpenAI Studio, it will automatically create all that for you. You end up having this kind of index structure:

Here:

• 'content' field has the processed document's main content
• 'contentVector' fied stores the embeddings of this main content (the output of a call to an 'text-embedding-ada-002' model with the same content which is stored in 'content' field)
• 'filepath' and 'url' have been filled with my blob file details
• 'title' has been created using the beginning of the document chunk... it's sometimes not a real title :D
• 'id' is a technical identifier in the search service
• 'chunk_id' is a zero-based index indicating which part of the document it is, because the input documents are split in littler chunks (see more details here)

I also want any new data added to my blob container to be automatically included in Azure AI Search.

You need to define the frequency of processing of your indexer to run regularly. Of you can use some code, for example by using an Azure Function with a "Blob trigger" event, where you will index your document when the function runs because it is fired when a blob is created / modified.