FoveatedStreamingSession headset and controller location smoothing, Velocity for OpenXR

Hi,

We build a PC VR streaming app for Apple Vision Pro. It has two VR paths: our own stream (our transport, VideoToolbox decode, our compositor layer) and FoveatedStreamingSession with CloudXR, which we use for titles that benefit from foveation. On our own path we filter the head and controller poses and send velocity. Getting both right took a lot of on-device tuning, and they made a large difference to comfort and to physics-based games. On the FoveatedStreamingSession path, the pose path from tracking to the endpoint runs inside the system, so an app has no way to apply, adjust or even inspect either one. We would like to ask for that control.

What we measured on device: Vision Pro tracking picks up very small, heartbeat-rate motion (around 1.2 Hz, millimetre scale). Locally that is invisible. After a streaming round trip it comes back as a visible world shake, because the remote frame is rendered at a pose about 50 ms old and reprojection has to cover the difference. Our A/B tests showed the tremor is largely rotational. Reprojection did not fully absorb it at our latency: turning off only the rotation filter brought the shake back right away.

What fixed it for us: a One-Euro filter on the head pose, applied on the client before the pose is sent, with the frame anchored at the exact pose it was rendered with, so reprojection still covers the gap to the real head at no added latency. The values we settled on after live tuning:

The right values depended on the content, the user and the end-to-end latency, which is why we are asking for control rather than one fixed default. Controllers need their own settings, because the right balance between lag and steadiness differs between fast games and precise work such as aiming or drawing.

What we would like on FoveatedStreamingSession:

• Separate smoothing settings for the head and for tracked accessories, with position and rotation set independently.
• A choice of system default, off, or custom parameters. A One-Euro style minimum cutoff plus speed coefficient would cover our case.
• Documentation of what filtering, if any, the system applies today, so apps don't filter on top of it.

Physics titles (throws, grab-and-pull, swings) take their motion from the linear and angular velocity in the OpenXR space velocity (XrSpaceVelocity), not from changes in position.

• If velocity arrives as zero, throws drop at the user's feet.
• If it arrives noisy, the PC runtime's own extrapolation to photon time amplifies the noise, and the controllers shake at rest.

We hit both cases on our own path. What fixed them:

1. Pass the accessory anchor's velocity through.
2. Set the pose time offset from the client's own prediction, so the PC side does not predict twice.
3. Apply a small deadband (about 0.05 m/s linear and 10°/s angular, matching common open-source streamers).

What we would like on FoveatedStreamingSession:

• Documentation of whether linear and angular velocity are sent to the endpoint for the head and for tracked accessories, and in which reference frame. We could not find a frame statement for AccessoryAnchor velocity in the docs or the SDK interface. Game title testing appears as though these values not being sent.
• App control over velocity: on/off, a gain, a deadband, and a filter (or the same smoothing settings as above).
• The prediction offset the session uses when it sends poses, so apps and endpoints can avoid predicting twice.

extension FoveatedStreamingSession {
    struct PoseFiltering: Sendable, Equatable {
        static var systemDefault: PoseFiltering { get }
        static var off: PoseFiltering { get }
        static func oneEuro(positionMinCutoff: Double, positionBeta: Double,
                            rotationMinCutoff: Double, rotationBeta: Double) -> PoseFiltering
    }

    struct VelocityReporting: Sendable, Equatable {
        var isEnabled: Bool
        var linearGain: Float
        var angularGain: Float
        var linearDeadband: Float   // m/s
        var angularDeadband: Float  // rad/s
        var filtering: PoseFiltering
    }

    var headPoseFiltering: PoseFiltering { get set }
    var accessoryPoseFiltering: PoseFiltering { get set }
    var accessoryVelocityReporting: VelocityReporting { get set }
    var posePredictionInterval: TimeInterval { get }
}

Being able to change these during a session (not only before connect) would let users tune them from the app's own settings while wearing the headset, which is how we found our values.