Audio remains mono after turning voiceProcessing off

Using AVAudioEngine with an AVAudioPlayerNode which plays stereo sound works perfectly. I even understand that turning setVoiceProcessingEnabled on the inputNode turns the sound mono. But after I stop the session and the engine, and turn voice processing off, the sound remains mono. This issue is only present with the built in speakers. This is what the I/O formats of the nodes look like before and after the voiceProcessing on-off: Before:

mainMixer input<AVAudioFormat 0x281896580: 2 ch, 44100 Hz, Float32, non-inter>
mainMixer output<AVAudioFormat 0x2818919f0: 2 ch, 44100 Hz, Float32, non-inter>
outputNode input<AVAudioFormat 0x281891cc0: 2 ch, 44100 Hz, Float32, non-inter>
outputNode output<AVAudioFormat 0x281891770: 2 ch, 44100 Hz, Float32, non-inter>

After:

mainMixer input<AVAudioFormat 0x2818acaf0: 2 ch, 44100 Hz, Float32, non-inter>
mainMixer output<AVAudioFormat 0x2818acaa0: 1 ch, 44100 Hz, Float32>
outputNode input<AVAudioFormat 0x281898820: 1 ch, 44100 Hz, Float32>
outputNode output<AVAudioFormat 0x2818958b0: 2 ch, 44100 Hz, Float32, non-inter>

Sadly just changing the connection type does not solve anything, I already tried that with (this solves the stereo issue on headphones, but not on built in speakers):

let format = AVAudioFormat(standardFormatWithSampleRate: 48000, channels: 2)!
audioEngine.connect(audioEngine.mainMixerNode, to: audioEngine.outputNode, format: format)