AVFoundation: Play Audio Delayed in Background with Ducking

I have a timer in one of my apps. I now want to add audio that plays at the end of the timer. It's a workout app and the sound should remind the user that it is time for the next exercise.

The audio should duck music playback (Apple Music / Spotify) and also work in background. Background audio is enabled for the app.

I am not able to achieve everything at the same time.

I set the audio session to category playback with options duckOthers.

do {
    try AVAudioSession.sharedInstance().setCategory(
        .playback,
        options: .duckOthers
)
} catch {
    print(error)
}

For playback I just use the AVAudioPlayer. When the user starts the timer, i schedule a timer in the future and play the sound.

While this works perfectly in the foreground, the sound is not played back when going to background, as timers are not fired in the background, but rather when the user puts the app back in foreground.

I have also tried using AVAudioEngine and AVAudioPlayerNode, as the latter can start playback delayed. The case from above works now, but the audio ducking begins immediately when initialising the AVAudioEngine, which is also not what i want.

Is there any other approach that I am not aware of?