dockeropenai-apiollamanixpacks

How to Connect a Dockerized Python App to Ollama Running on Host (localhost:11434) – Connection Error


I'm running Ollama locally on my machine with the llama3.2:latest model. I can confirm the model is up and running:

ollama ps
NAME ID SIZE PROCESSOR UNTIL
llama3.2:latest a80c4f17acd5 4.0 GB 100% GPU 3 minutes from now

Ollama is accessible at http://localhost:11434/.

I have a Python script that uses the pydantic_ai and OpenAIProvider to connect to Ollama via its OpenAI-compatible API. When I run this script directly on my development machine, it works as expected:

from pydantic import BaseModel
from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIModel
from pydantic_ai.providers.openai import OpenAIProvider

class CityLocation(BaseModel):
    city: str
    country: str

ollama_model = OpenAIModel(
    model_name='llama3.2', provider=OpenAIProvider(base_url='http://localhost:11434/v1')
)
agent = Agent(ollama_model, output_type=CityLocation)

result = agent.run_sync('Where were the olympics held in 2012?')
print(result.output)
print(result.usage())

Now, I want to run this code inside a Docker container. I built the image using Nixpacks:

sudo nixpacks build . -t image1

It works and tell me to run :

  docker run -it b0300949-bf2f-404e-9ea1-ebaa34423b50

I then run the image with the same command:

But it fails. I have a very long error message ending like that:

openai.APIConnectionError: Connection error

I also tried running the same command with --network="host":

But I still get the same connection error.

Questions:

How can I get my Docker container to successfully connect to Ollama running on my host at http://localhost:11434/v1?


Solution

  • 1)first step is probably not necessary for you. But for me Ollama run on windows und the container on WSL..
    In %UserProfile% I have written that:

    [wsl2]

    networkingMode=mirrored

    Is I don^t do that WSL and windows have a different localhost

    2)

    my code :

    from pydantic import BaseModel
    
    from pydantic_ai import Agent
    from pydantic_ai.models.openai import OpenAIModel
    from pydantic_ai.providers.openai import OpenAIProvider
    
    import requests
    try:
    
        response = requests.get("http://127.0.0.1:11434")
        #response = requests.get("http://localhost:11434") #equivalent
        print("Status code:", response.status_code)
        print("Response body:", response.text)
    except requests.exceptions.RequestException as e:
        print("Connection failed:", e)
    
    # exit()
    
    class CityLocation(BaseModel):
        city: str
        country: str
    
    ollama_model = OpenAIModel(
        model_name='llama3.2', 
        provider=OpenAIProvider(base_url="http://127.0.0.1:11434/v1")
        #provider=OpenAIProvider(base_url='http://localhost:11434/v1') #equivalent
    )
    agent = Agent(ollama_model, output_type=CityLocation)
    
    result = agent.run_sync('Where were the olympics held in 2012?')
    print(result.output)
    #> city='London' country='United Kingdom'
    print(result.usage())
    """
    Usage(requests=1, request_tokens=57, response_tokens=8, total_tokens=65, details=None)
    """
    
    1. How the image is created:
    sudo nixpacks build . -t image1
    
    1. how the image is launch:
     sudo docker run -it --network host image1
    
    1. the result (it works)

    Status code: 200

    Response body: Ollama is running

    city='London' country='United Kingdom'

    Usage(requests=1, request_tokens=176, response_tokens=18, total_tokens=194, details=None)