Override value in pydantic model with environment variable

I am building some configuration logic for a Python 3 app, and trying to use pydantic and pydantic-settings to manage validation etc. I'm able to load raw settings from a YAML file and create my settings object from them. I'm also able to read a value from an environment variable. But I can't figure out how to make the environment variable value take precedence over the raw settings:

import os
import yaml as pyyaml
from pydantic_settings import BaseSettings, SettingsConfigDict


class FooSettings(BaseSettings):
    foo: int
    bar: str
    model_config = SettingsConfigDict(env_prefix='FOOCFG__')


raw_yaml = """
foo: 13
bar: baz
"""

os.environ.setdefault("FOOCFG__FOO", "42")

raw_settings = pyyaml.safe_load(raw_yaml)
settings = FooSettings(**raw_settings)

assert settings.foo == 42

If I comment out foo: 13 in the input yaml, the assertion passes. How can I make the env value take precedence?

Solution

Are you sure you want the environment to take precedence? While not ubiquitous, it is very common for environment variables to have the lowest precedence (typically, the ordering is built-in defaults, then environment variables, then configuration files, then command line options). Deviating from this convention can be surprising.

You could get the behavior you want for a specific field by adding a field validator that checks for the appropriate environment variable and uses that value in preference to an existing value if it is avaiable. Something like:

import os
import yaml as pyyaml
from pydantic_settings import BaseSettings, SettingsConfigDict
from pydantic import field_validator


class FooSettings(BaseSettings):
    model_config = SettingsConfigDict(env_prefix="FOOCFG__")

    foo: int
    bar: str

    @field_validator("foo", mode="after")
    @classmethod
    def validate_foo(cls, val):
        '''Always use the value from the environment if it's available.'''
        if env_val := os.environ.get(f"{cls.model_config['env_prefix']}FOO"):
            return int(env_val)

        return val


raw_yaml = """
foo: 13
bar: baz
"""

os.environ.setdefault("FOOCFG__FOO", "42")

raw_settings = pyyaml.safe_load(raw_yaml)
settings = FooSettings(**raw_settings)

assert settings.foo == 42

If you wanted to do this for all fields, you could use a model validator instead. Maybe something like this?

    @model_validator(mode="after")
    @classmethod
    def validate_foo(cls, data):
        for field in cls.model_fields:
            env_name = f'{cls.model_config["env_prefix"]}{field.upper()}'
            if env_val := os.environ.get(env_name):
                setattr(data, field, type(getattr(data, field))(env_val))

        return data