I'm shipping a PCVR streaming app using PSVR2 Sense controllers on the Vision Pro. Four pain points I'd love clarity on:
- PSVR2 L3/R3 thumbstick click at rest — Apple's GameController framework doesn't fire a stick-click event unless there's axis deflection. Users physically clicking the stick straight in get nothing. Is this an intentional filter, and is a change planned?
- Haptic forwarding PC → AVP. On CloudXRKit there's NSKStreamClientHapticDelegate for dispatching server-generated haptic events into CHHapticEngine. Does FoveatedStreaming have an equivalent delegate, or are streamer apps expected to bring their own side-channel for haptics?
- Controller pose — we're currently going through ARKit's AccessoryTrackingProvider and combining with GCController input. Is that the recommended supported path for FoveatedStreaming apps, or is there something more native to the framework planned?
Gaze clarity :-
- The framework already uses gaze internally to drive foveation tile selection on the server. Is any per-frame metadata returned to the client that tells us — even loosely — where the user is looking? A gaze quadrant, a foveal-tile coordinate, distance-from-fovea-centre. Anything we can feed back to the PC to tune its own pipeline.
- Privacy-preserving aggregates — would Apple consider exposing fixation/saccade ratios, gaze stability over the last N frames, dwell-on-region signals? Lets apps push higher detail
without ever learning raw direction. 3. What's the actual barrier — strictly privacy, or also performance? If it's privacy, would an on-device processing surface (App Intent style, Background Asset extension, etc.) be a viable path to let apps act on gaze without raw access? 4. are there public APIs in this area today that I've missed? I've been working from the swiftinterface + WWDC sessions and don't want to be reinventing something that already exists in a quieter corner of the SDK.
Hello!
PSVR2 L3/R3 thumbstick click at rest — Apple's GameController framework doesn't fire a stick-click event unless there's axis deflection. Users physically clicking the stick straight in get nothing. Is this an intentional filter, and is a change planned?
This sounds like a bug. Please file a feedback item via feedback assistant. I'm curious if you are seeing this specifically with the FoveatedStreaming framework, or also when using GameController framework more generally?
Haptic forwarding PC → AVP. On CloudXRKit there's NSKStreamClientHapticDelegate for dispatching server-generated haptic events into CHHapticEngine. Does FoveatedStreaming have an equivalent delegate, or are streamer apps expected to bring their own side-channel for haptics?
This is a known issue with the framework on 26.4 and 27.0 beta 1, and something the team is looking into.
Controller pose — we're currently going through ARKit's AccessoryTrackingProvider and combining with GCController input. Is that the recommended supported path for FoveatedStreaming apps, or is there something more native to the framework planned?
This is the correct path if you are rolling your own pose + controller input.
The framework already uses gaze internally to drive foveation tile selection on the server. Is any per-frame metadata returned to the client that tells us — even loosely — where the user is looking? A gaze quadrant, a foveal-tile coordinate, distance-from-fovea-centre. Anything we can feed back to the PC to tune its own pipeline.
Privacy-preserving aggregates — would Apple consider exposing fixation/saccade ratios, gaze stability over the last N frames, dwell-on-region signals? Lets apps push higher detail
Eye input data is restricted to the foveated streaming protocol. This is intentional to help protect user privacy.
are there public APIs in this area today that I've missed? I've been working from the swiftinterface + WWDC sessions and don't want to be reinventing something that already exists in a quieter corner of the SDK.
Yes - good news! We have shipped a new API this year, which allows streaming providers (separate from NVIDIA CloudXR) to extend visionOS with their own Foveated Streaming Protocol. These streaming providers are granted access to approximate eye input data, in a separate extension process, to facilitate the implementation of foveated streaming. Please see my post here for more information.
