Spatial audio

So I updated to ios14 I wanted to try spatial audio with my AirPods Pro. It works 50/50 like when I move my head it picks it up and changes where the audio comes from but when I move my phone it doesn’t pick it up I’m on a 7 plus both devices are on the latest firmware. Do let me know now what I can do. Btw I don’t use Apple TV I was watching the Apple event on the Apple TV app.