As in the question title, what's the relationship between NSAppearance, NSEffectView.Material, and "vibrancy"? I've found through experimentation that, for some materials, the choice of NSAppearance can change how the material appears (e.g. NSEffectView.Material.titlebar will be light or dark depending on the active NSAppearance), while other materials (e.g. .light) don't seem to care.
I suspect that materials like .titlebar are proxies which select from .dark, .ultradark, .light, and .mediumLight depending on the NSAppearance, but then that would seem to be the role of .appearanceBased. I also see in the description for NSAppearance.Name.vibrantLight...
This should only be set on an NSVisualEffectView or one of its subviews.
...which somewhat contradicts a statement from the NSEffectView documentation...
The view’s effective appearance must allow vibrancy... in most cases you set the appearance on the window or on the visual effect view—subviews then inherit the appearance.
...suggesting that it could be correct to set vibrantLight as the NSAppearance of an entire window (if that's the look you wanted).
Finally, I'm confused as to what exactly "vibrancy" is; if someone could explain it, that would be great.
So an NSAppearance generally describes the styling of controls, colors, etc for a view hierarchy that the appearance is set against.
NSVisualEffectView provides a way to achieve two effects: translucency and vibrancy. The former is the more obvious, with the translucent sidebars or titlebars. And the documentation has a really nice description of vibrancy:
Vibrancy is associated with translucency. It describes a compositing mode that does special blending such as Plus Lighter, Plus Darker, Color Dodge, or Color Burn.
Basically describing how the content (text, images, etc) within the visual effect view are composited against the translucency.
So how do these all relate?
Material describes the look of the translucency effect. As you pointed out, some are affected by NSAppearance, some are not. The ones that are semantically describe their usage so that custom UI can resemble that effect regardless of appearance (.appearanceBased, .titlebar, .menu, .popover, .sidebar, .selection) whereas others allow for specific control over the resulting translucency (.light, .dark, .mediumLight, .ultraDark) but should be used in conjunction with their associated NSAppearance so that the content within the visual effect view can match the translucency effect. Unless you need specific control over the material, using appearance-sensitive / semantic ones can result for more standard UI.
So in order to get the content vibrancy effects that NSVisualEffectView can provide, it needs to be used in conjunction with a vibrant appearance: .vibrantLight or .vibrantDark. Without setting a "vibrant" appearance, NSVisualEffectView will only provide the translucency effect in the background, and the content within it will look plain and not have the special blending modes like you see in sidebars or titlebars.