Edit - I eventually figured out the answer and have posted it below.
Using .audio_set_volume() on a media_player_new() object works fine with values 0-100, but it's much quieter than the corresponding value in normal VLC is, by a factor of around 2-3. This can be remedied by using values greater than 100, but this introduces the problem of severe delays while changing the volume (not delays in the video or audio, just ~half-second delays before the volume updates).
No issues with my volume mixing from what I can tell. The player is being embedded in PyQt5. I can't find anyone else with this issue so I imagine there's an easy workaround I'm missing.
I never got a response, but I eventually figured it out on my own: Firstly, running it through a command prompt/natively through Python causes the volume to be lower than normal (no idea why). This goes away when compiling or using your script as a default program.
Secondly, there's a VLC command-line argument called --gain you can set that seems to default to a lower value when using libvlc directly versus what VLC actually uses. When defining your instance, specify the argument like so (it takes a float value from 0-8):
instance = vlc.Instance(['--gain=8.0']) # make sure the arguments are in a list!
A gain of 8.0 is definitely higher than what VLC natively uses, but it's not ear-shatteringly loud. From what I can tell, the quality is not degraded at all and there's no delay while adjusting the volume with --gain set.
Don't forget to include any other arguments in the list if desired, such as ones from sys.argv.