Unable to set AudioSession inactive with AVPlayer

I am using an AVPlayer to playback audio in my iOS app (because the audio files I'm playing are not local), and when my track finishes, I want to hide the InfoCenter controls, but I get this error:

AVAudioSession.mm:1079:-[AVAudioSession setActive:withOptions:error:]: Deactivating an audio session that has running I/O. All I/O should be stopped or paused prior to deactivating the audio session.

I am however pausing my AVPlayer before.

This is my code:

player.pause()
player.replaceCurrentItem(with: nil)

MPNowPlayingInfoCenter.default().nowPlayingInfo = nil
try? audioSession.setActive(false, options: .notifyOthersOnDeactivation)

Due to this error, the native controls remain visible, but don't work anymore.

A quick way to lose the lock screen controls is to make your audio session mixable (without deactivating your audio session):

try! AVAudioSession.sharedInstance().setCategory(.playback, mode: .default, options: .mixWithOthers)

Remove the .mixWithOthers when you want them back.

Another, perhaps more palatable way is to remove your MPRemoteCommandCenter targets once you pause playback. e.g. if you set it up like this

var playTarget: Any? = nil
var pauseTarget: Any? = nil

func addRemoteTargets() {
    let commandCenter = MPRemoteCommandCenter.shared()

    playTarget = commandCenter.playCommand.addTarget { (event) -> MPRemoteCommandHandlerStatus in
        print("play \(event)")
        return .success
    }

    pauseTarget = commandCenter.pauseCommand.addTarget { (event) -> MPRemoteCommandHandlerStatus in
        print("pause \(event)")
        return .success
    }
}

Then remove the targets like so:

func removeRemoteTargets() {
    let commandCenter = MPRemoteCommandCenter.shared()
    commandCenter.playCommand.removeTarget(playTarget)
    playTarget = nil
    commandCenter.pauseCommand.removeTarget(pauseTarget)
    pauseTarget = nil
}

p.s. these are "seems to work!" answers. I don't know how you are supposed to determine when an AVPlayer has finished its audio I/O (apart from polling via setActive:false failing). My reasoning is that becoming the lock screen/"Now Playing App" has 3 pieces (this is sort of documented somewhere, this is the closest thing I can find to that right now, although the most explicit documentation of how this works is in the system logs):

1. a non mixable audio session category
2. MPRemoteCommandCenter integration
3. current or "recently" finished audio IO

so if you can remove any one of these pieces, the lock screen controls should go away.