I wonder how do OSes store the keys used to encrypt things like website passwords. After searching online I found only two answers but they only refer to how service providers should store passwords (they should store a hash and a salt), or how apps should store them (they should use system APIs that provide encryption).
But what I want to know is how the system itself protects the encryption keys. I know some devices have special hardware to store some important keys (like the Secure Enclave in Apple chips). But I want to know how it would be done in a regular computer.
I mean they probably store it somewhere on the disk, but how do they do it securely so you can't just read it?
The way almost every “secure credential storage” does it is by encrypting the credential storage file on disk with a random key further encrypted with a passphrase. Without the passphrase, you can’t gain access to the key needed to decrypt the contents. You can also store the key in a crypto device (e.g. a NFC smart card or a USB dongle), and those would be protected by their own challenge system (passphrase, fingerprint, etc.).
The passphrase is obtained from the user directly by the OS elements, so that spoofing it (e.g. by a website) will be harder, and also so that no application needs direct access to the credential storage – once the user unlocks the storage, the application can obtain the credentials it originally stored, and only those. There are various mechanisms available for that, but a common one is using the application id of the executable that issues the query – with further limitation that the executable must be running only signed binary elements (no library injection etc), and not be subject to debugging (the OS has full awareness of that).
As for how do the OSes prevent the leakage of cleartext material while it remains in memory? Hah. That’s a whole another story, and there is a multitude of approaches – preventing the data from spilling to swap, using encrypted swap, decrypting only the credentials immediately needed and discarding them as soon as they are not needed, involving the kernel in managing the online storage (in RAM) while the system is running, and using the kernel to run all cryptographic operation that need access to e.g. asymmetric cipher private keys. The kernel can further relegate this to a hardware device, while the OS provides a unified API no matter where the private keys are stored. Such APIs exist on OS X and Windows, and surely on Linux too.
Since the TPM or USB devices like crypto card readers can securely store the private keys, it is really useful to use personal certificates for authentication, instead of passwords. The device with the private key can then sign a challenge from the remote server (e.g. certificate-supporting website), and the remote system checks the challenge’s signature against your public and thus not sensitive certificate. Side rant here: I wish financial institution and everyday websites would support personal certificates instead of passwords. It is more secure and completely hassle-free, and works across all platforms – even mobile!