I'm running Ollama locally on my machine with the llama3.2:latest model. I can confirm the model is up and running:
ollama ps
NAME | ID | SIZE | PROCESSOR | UNTIL |
---|---|---|---|---|
llama3.2:latest | a80c4f17acd5 | 4.0 GB | 100% GPU | 3 minutes from now |
Ollama is accessible at http://localhost:11434/.
I have a Python script that uses the pydantic_ai and OpenAIProvider to connect to Ollama via its OpenAI-compatible API. When I run this script directly on my development machine, it works as expected:
from pydantic import BaseModel
from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIModel
from pydantic_ai.providers.openai import OpenAIProvider
class CityLocation(BaseModel):
city: str
country: str
ollama_model = OpenAIModel(
model_name='llama3.2', provider=OpenAIProvider(base_url='http://localhost:11434/v1')
)
agent = Agent(ollama_model, output_type=CityLocation)
result = agent.run_sync('Where were the olympics held in 2012?')
print(result.output)
print(result.usage())
Now, I want to run this code inside a Docker container. I built the image using Nixpacks:
sudo nixpacks build . -t image1
It works and tell me to run :
docker run -it b0300949-bf2f-404e-9ea1-ebaa34423b50
I then run the image with the same command:
But it fails. I have a very long error message ending like that:
openai.APIConnectionError: Connection error
I also tried running the same command with --network="host":
But I still get the same connection error.
Questions:
How can I get my Docker container to successfully connect to Ollama running on my host at http://localhost:11434/v1?
1)first step is probably not necessary for you. But for me Ollama run on windows und the container on WSL..
In %UserProfile% I have written that:
[wsl2]
networkingMode=mirrored
Is I don^t do that WSL and windows have a different localhost
2)
my code :
from pydantic import BaseModel
from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIModel
from pydantic_ai.providers.openai import OpenAIProvider
import requests
try:
response = requests.get("http://127.0.0.1:11434")
#response = requests.get("http://localhost:11434") #equivalent
print("Status code:", response.status_code)
print("Response body:", response.text)
except requests.exceptions.RequestException as e:
print("Connection failed:", e)
# exit()
class CityLocation(BaseModel):
city: str
country: str
ollama_model = OpenAIModel(
model_name='llama3.2',
provider=OpenAIProvider(base_url="http://127.0.0.1:11434/v1")
#provider=OpenAIProvider(base_url='http://localhost:11434/v1') #equivalent
)
agent = Agent(ollama_model, output_type=CityLocation)
result = agent.run_sync('Where were the olympics held in 2012?')
print(result.output)
#> city='London' country='United Kingdom'
print(result.usage())
"""
Usage(requests=1, request_tokens=57, response_tokens=8, total_tokens=65, details=None)
"""
sudo nixpacks build . -t image1
sudo docker run -it --network host image1
- the result (it works)
Status code: 200
Response body: Ollama is running
city='London' country='United Kingdom'
Usage(requests=1, request_tokens=176, response_tokens=18, total_tokens=194, details=None)