Excel in Large-Scale Data Processing with GPT-3.5 and Embeddings

I'm working on integrating OpenAI functionalities, specifically GPT3.5 and embeddings, into a large system of Excel workbooks used for almost anything in my office. Our goal is having GPT3.5 taking over the place of a set of macros that has become hell maintaining, and the embeddings-ada-002 for improving the searches and categorization of data.

However, I'm concerned of Excel being capable to handle hundreds of these high-dimensional embeddings. I'm also afraid that Excel and GPT3.5 will struggle when having multiple sheets of thousands of rows tall and "complex structures" provided as a context.

(We run Windows on i5 processors)

Will Excel be able to handle such large-scale data processing with AI models? Any insights on this matter?

Excel is awesome but it has its limitations when it comes to handling large volumes of data. It is designed for tabular data and it will struggle with the high-dimensional data Source.

Also GPT-3.5 has limitations of the number of tokens it can handle Source. If your Excel sheets is data heavy, GPT-3.5 will struggle.

A more scalable solution could involve using Python. You could use Python with pandas for data manipulation and PyOpenAI to read data from Excel workbooks, process it using the OpenAI models, and then write the results back into Excel Source.

Here's a example:

import pandas as pd
from openai import OpenAI, GPT3Completion

# Load your data from Excel into a pandas DataFrame
df = pd.read_excel('your_data.xlsx')

# Initialize the OpenAI API with your API key
openai = OpenAI(api_key='your_api_key')

# Process each row of your data using GPT3.5
for index, row in df.iterrows():
    prompt = row['your_column']
    response = openai.GPT3Completion.create(prompt=prompt, model='gpt-3.5-turbo')
    # Save logic