I have an app that is largely configured by environment variables. I use Pydantic Settings to define the settings available, and validate them. I have an initial set of settings, and the regular app settings.
The initial settings are ones that should not fail validation, and contain essential settings for starting the app.
For example, when my app starts up, if the regular Settings() can't be initialized because something in them failed validation, I still want to be able to send the error to Sentry. For that, I need SENTRY_DSN to configure Sentry. SENTRY_DSN can't be part of the regular settings, because if something unrelated in Settings fails validation, I won't have access to SENTRY_DNS either.
Right now, my settings look like this:
class InitialSettings(BaseSettings):
model_config = SettingsConfigDict(
env_file="settings.env",
env_file_encoding="utf-8",
extra="ignore",
env_ignore_empty=True,
env_nested_delimiter="__",
case_sensitive=True,
)
SENTRY_DSN: Annotated[
Optional[str],
Field(None),
]
class Settings(BaseSettings):
model_config = SettingsConfigDict(
env_file="settings.env",
env_file_encoding="utf-8",
extra="ignore",
env_ignore_empty=True,
env_nested_delimiter="__",
case_sensitive=True,
)
STORAGE: Annotated[
LocalStorageSettings | S3StorageSettings,
Field(..., discriminator="STORAGE_TYPE"),
]
DEBUG: Annotated[DebugSettings, Field(default_factory=DebugSettings)]
...
This works. When my app starts up, I first initialize InitialSettings(), and then try to initialize Settings(). If Settings() fails, I can still use the SENTRY_DSN setting to send the error to Sentry.
The issues comes when I try to have both settings use the same env file (settings.env), AND enable the extra="forbid" feature on Settings().
I like the idea of having extra="forbid" enabled, but that also means that if I enable it on Settings(), it will always fail, because the env file will contain an entry for SENTRY_DSN, which Settings doesn't know about.
To fix, this I tried to add InitialSettings to Settings like this:
class Settings(BaseSettings):
model_config = SettingsConfigDict(
env_file="settings.env",
env_file_encoding="utf-8",
extra="forbid",
env_ignore_empty=True,
env_nested_delimiter="__",
case_sensitive=True,
)
STORAGE: Annotated[
LocalStorageSettings | S3StorageSettings,
Field(..., discriminator="STORAGE_TYPE"),
]
DEBUG: Annotated[DebugSettings, Field(default_factory=DebugSettings)]
INIT: Annotated[InitialSettings, Field(default_factory=InitialSettings)]
...
Now Settings should know about all the settings defined in InitialSettings, and if there's any extra settings in the env file that aren't defined in either class, it should fail.
This almost works.
The problem is that when you call InitialSettings, the SENTRY_DSN in the env file is expected to just be called SENTRY_DSN. When you call InitialSettings, because InitialSettings is nested under INIT, it expects the sentry variable to be called INIT__SENTRY_DSN.
How do I configure Pydantic Settings so that all settings under InitialSettings always look for SENTRY_DSN, no matter if they are initialized using InitialSettings(), or Settings()?
Note: I still want the other nested settings classes under Settings, like STORAGE, to work the same - be prefixed with STORAGE__ in the env file.
The simplest solution is to use two different .env files: one for InitialSettings and one for Settings:
# initial.env
SENTRY_DSN=...
# settings.env
# your other `Settings` envs here without `SENTRY_DSN`
class InitialSettings(BaseSettings):
model_config = SettingsConfigDict(
env_file="initial.env",
env_file_encoding="utf-8",
extra="ignore",
env_ignore_empty=True,
env_nested_delimiter="__",
case_sensitive=True,
)
SENTRY_DSN: Annotated[
Optional[str],
Field(None),
]
class Settings(CustomSettings):
model_config = SettingsConfigDict(
env_file="settings.env",
env_file_encoding="utf-8",
extra="forbid",
env_ignore_empty=True,
env_nested_delimiter="__",
case_sensitive=True,
)
STORAGE: Annotated[
LocalStorageSettings | S3StorageSettings,
Field(..., discriminator="STORAGE_TYPE"),
]
DEBUG: Annotated[DebugSettings, Field(default_factory=DebugSettings)]
INIT: Annotated[InitialSettings, Field(default_factory=InitialSettings)]
...
Using two separate files with different environment variables in your case seems logical. You have two independent classes of settings, with different extra settings, so different validation will be performed. In Settings extra environment variables will throw a validation error, while InitialSettings will not. But, most importantly, using a separate file with environment variables will allow InitialSettings to initialize correctly both inside the Settings class and separately.
Further, I'm assuming that you can't use multiple .env files, for whatever reason, and that you plan to leave the setting - extra=“forbid” in Settings. Based on this, I can suggest two solutions.
The first solution is to use the INIT prefix in your settings.env for SENTRY_DSN. This would look something like this:
# setting.env
INIT__SENTRY_DSN=...
# other envs
Then your settings classes will look like this:
class InitialSettings(BaseSettings):
model_config = SettingsConfigDict(
env_prefix="INIT__",
env_file="settings.env",
env_file_encoding="utf-8",
extra="ignore",
env_ignore_empty=True,
env_nested_delimiter="__",
case_sensitive=True,
)
SENTRY_DSN: Annotated[
Optional[str],
Field(None),
]
class Settings(BaseSettings):
model_config = SettingsConfigDict(
env_file="settings.env",
env_file_encoding="utf-8",
extra="forbid",
env_ignore_empty=True,
env_nested_delimiter="__",
case_sensitive=True,
)
STORAGE: Annotated[
LocalStorageSettings | S3StorageSettings,
Field(..., discriminator="STORAGE_TYPE"),
]
DEBUG: Annotated[DebugSettings, Field(default_factory=DebugSettings)]
INIT: Annotated[InitialSettings, Field(default_factory=InitialSettings)]
...
In this case, because we set SettingsConfigDict(env_prefix=“INIT__”, ...) in InitialSettings, both classes of settings: will look for one environment variable INIT__SENTRY_DSN in the settings.env file. Both classes will properly initialize SENTRY_DSN from INIT__SENTRY_DSN.
Another possible solution could be to override settings_customize_sources by creating a subclass of BaseSettings and add its own DotEnvSettingsSource handler:
# setting.env
SENTRY_DSN=...
# other envs
from typing import Annotated, Optional
from pydantic import Field
from pydantic_settings import (
BaseSettings,
DotEnvSettingsSource,
PydanticBaseSettingsSource,
SettingsConfigDict,
)
class CustomDotEnvSettingsSource(DotEnvSettingsSource):
def __call__(self):
# This method may actually look simpler, since `InitialSettings`
# can be initialized using an `settings.env` file
# (but I like the option below better, it's more explicit):
# def __call__(self):
# result = super().__call__()
# result.pop('SENTRY_DSN', None)
# return result
result = super().__call__()
if 'SENTRY_DSN' in result:
value = result.pop('SENTRY_DSN')
result.setdefault('INIT', {})['SENTRY_DSN'] = value
return result
class CustomSettings(BaseSettings):
@classmethod
def settings_customise_sources(
cls,
settings_cls: type[BaseSettings],
init_settings: PydanticBaseSettingsSource,
env_settings: PydanticBaseSettingsSource,
dotenv_settings: PydanticBaseSettingsSource,
file_secret_settings: PydanticBaseSettingsSource,
) -> tuple[PydanticBaseSettingsSource, ...]:
return super().settings_customise_sources(
settings_cls=settings_cls,
init_settings=init_settings,
env_settings=env_settings,
# register custom `DotEnvSettingsSource` handler
dotenv_settings=CustomDotEnvSettingsSource(settings_cls),
file_secret_settings=file_secret_settings,
)
class InitialSettings(BaseSettings):
model_config = SettingsConfigDict(
env_file="settings.env",
env_file_encoding="utf-8",
extra="ignore",
env_ignore_empty=True,
env_nested_delimiter="__",
case_sensitive=True,
)
SENTRY_DSN: Annotated[
Optional[str],
Field(None),
]
class Settings(CustomSettings):
model_config = SettingsConfigDict(
env_file="settings.env",
env_file_encoding="utf-8",
extra="forbid",
env_ignore_empty=True,
env_nested_delimiter="__",
case_sensitive=True,
)
STORAGE: Annotated[
LocalStorageSettings | S3StorageSettings,
Field(..., discriminator="STORAGE_TYPE"),
]
DEBUG: Annotated[DebugSettings, Field(default_factory=DebugSettings)]
INIT: Annotated[InitialSettings, Field(default_factory=InitialSettings)]
...
All three methods described in this answer have been tested and should work for you.