The spec for blend-mode saturation says:

Creates a color with the saturation of the source color and the hue and luminosity of the backdrop color.

Originally I assumed it would be HSL as that's the only colorspace you can use in web development that has a saturation channel, but that's clearly not it:

.color {
  width: 100px;
  height: 100px;
}

.expected-color {
  background: hsl(270, 25%, 50%);
}

.backdrop-color {
  background: hsl(270, 85%, 50%);
}

.source-color {
  mix-blend-mode: saturation;
  background: hsl(45, 25%, 50%);
}

Expected
<div class="color expected-color"></div>

Actual
<div class="color backdrop-color">
  <div class="color source-color"></div>
</div>

It's not so easy to demonstrate HSV/HSB, but the testing I did using color converters (eg. link) showed that it's not that color space either.

.color {
  width: 100px;
  height: 100px;
}

.expected-color {
  /* HSB/HSV - 270, 25, 50 */
  background: #706080;
}

.backdrop-color {
  /* HSB/HSV - 270, 85, 50 */
  background: #491380;
}

.source-color {
  /* HSB/HSV - 45, 25, 50 */
  mix-blend-mode: saturation;
  background: #807860;
}

Expected
<div class="color expected-color"></div>

Actual
<div class="color backdrop-color">
  <div class="color source-color"></div>
</div>

Likewise with LCH (converter) which doesn't have a saturation channel but it's chroma channel is practically an analog for saturation:

.color {
  width: 100px;
  height: 100px;
}

.expected-color {
  /* LCH - 50, 25, 270 */
  background: rgb(90,121,161);
}

.backdrop-color {
  /* LCH - 50, 85, 270 */
  background: rgb(0,125,255);
}

.source-color {
  /* LCH - 50, 25, 45 */
  mix-blend-mode: saturation;
  background: rgb(157,107,90);
}

Expected
<div class="color expected-color"></div>

Actual
<div class="color backdrop-color">
  <div class="color source-color"></div>
</div>

Seems to be consistent across Edge, Chrome, and Firefox.

So what colorspace is mix-blend-mode: saturation defined in? Or am I misunderstanding how it works?

I did a bit (a lot) more digging, and it looks like the math they're using is some weird combination of disparate colorspaces combined with general color theory:

The function for saturation described in the spec is:

B(Cb, Cs) = SetLum(SetSat(Cb, Sat(Cs)), Lum(Cb))

Where Cb is the backdrop color, and Cs is the Source color; both in sRGB space (just means R,G, and B are in the range [0 - 1]).

the functions used within this are defined as:

    ClipColor(C)
        L = Lum(C)
        n = min(Cred, Cgreen, Cblue)
        x = max(Cred, Cgreen, Cblue)
        if(n < 0)
            C = L + (((C - L) × L) / (L - n))

        if(x > 1)
            C = L + (((C - L) × (1 - L)) / (x - L))

        return C

    Lum(C) = 0.3 x Cred + 0.59 x Cgreen + 0.11 x Cblue

    SetLum(C, l)
        d = l - Lum(C)
        Cred = Cred + d
        Cgreen = Cgreen + d
        Cblue = Cblue + d
        return ClipColor(C)

    Sat(C) = max(Cred, Cgreen, Cblue) - min(Cred, Cgreen, Cblue)

    SetSat(C, s)
        if(Cmax > Cmin)
            Cmid = (((Cmid - Cmin) x s) / (Cmax - Cmin))
            Cmax = s
        else
            Cmid = Cmax = 0
        Cmin = 0
        return C;

We can ignoring ClipColor and the SetX functions, as we're really just interested in determining the colorspace(s) (if any). Which leaves:

Lum(C) = 0.3 x Cred + 0.59 x Cgreen + 0.11 x Cblue

Sat(C) = max(Cred, Cgreen, Cblue) - min(Cred, Cgreen, Cblue)

What is Lum?

The name of the function and it's usage might have you assume that they're referring to the Luminance that is sometimes ascribed to the L in HSL, or the B or V in HSB/HSV. They're not. This function is used to determine the relative brightness of a color to the human eye. It comes from the YIQ colorspace:

Notice that the first row - which will map to the Y (Luma) value - is a perfect match (albeit, slightly more precise) with the magic numbers in the Lum(C) formula.

To contrast this:

the B/V channel in HSB/HSV is Max(R,G,B).

the L channel in HSL is ( Max(R,G,B) - Min{R,G,B) ) / 2.

What is Sat?

This apparently IS actually Saturation (of a sorts), but it's not from any particular colorspace. It's more an abstraction of how you might think about saturation in the RGB colorspace. The first suggestion that this be a way to think about saturation seems to come from this 2003 paper, which is largely a critique of the existing colorspaces of it's time, and a suggestion to use a since unused colorspace IHLS.

For contrast:

S in HSB/HSV is ( Max(R,G,B) - Min(R,G,B) ) / Max(R,G,B) or 0 if Max(R,G,B) is 0.

S in HSL is ( Max(R,G,B) - L ) / Min(L, 1 - L) or 0 if L is 1 or 0. (The same L as defined above).

The only conclusion I can draw from all this is that mix-blend-mode: saturation has nothing to do with colorspaces at all, and is more to do with "saturation" in a more general sense; and in much the same way mix-blend-mode: luminosity isn't necessarily about any particular colorspace so much it's as a way of retaining "the perception of brightness".

If we were to point a finger at the most likely candidate, it'd be the YIQ colorspace, which I hadn't heard of before delving into this. The oversimplification of Saturation used in the spec can be referred to as a measure of Chrominance which is what the I and Q channels collectively represent.

It seems that the naming of these blend-modes is unavoidably ambiguous to a software developer, as their intended audience is likely designers, not engineers.

Not the answer I wanted, but it does seem to be the answer.

Update

I've just had an interesting exchange with one of the editors (authors?) of the spec. The justification for this mess is that (paraphrasing) "Adobe products have been doing it this way for the last 20 years and that's what people are used to". There's obviously no point interrogating that poor strain of reasoning too much, but I get the impression that the spec is largely a copy-paste of the existing blend modes implemented in photoshop and the like.