Hello there, I have a question and a potential solution to how these data could be opened to people developing streamers for foveated encoding.
Is there any plans to open up any type of gaze data, be it exact or just a general area instead of a pinpoint?
And if so I have a solution in mind, There could be a program for developers who are specifically making streaming apps in which Apple can come in and take a look at the streaming app and make a judgment from there if the developer would be trustworthy to allow access to gaze data.
This would solve any privacy concerns regarding data scraping if the app was looked through beforehand while still enabling access to the needed data for foveated streaming.
And beforehand there could be a prompt asking if you would like to allow the app to access gaze data or not much like other solutions.
More developers of VR streaming apps having access to this would be a net positive thing as it would create more robust options to stream VR to The vision pro, as cloud XR is relatively limited in terms of what it supports. In turn it would create more incentive for more developers to develop their streaming solutions for the vision pro.
Thank you for hosting this q&a!
Hello, @WhisperTheWolf --
Good news! We have added functionality very similar to what you propose in visionOS 27.0. It's called FoveatedStreamingProvider.
FoveatedStreamingProvider is a way to extend visionOS with a streaming protocol that is different from the built-in default protocol (NVIDIA CloudXR).
Here's how it works:
- Streaming protocol developers ship an app on the App Store. This app includes an
appex(ExtensionKit extension) that extends visionOS with the new foveated streaming protocol. These app extensions implementFoveatedStreamingProviderand, after review from Apple, are granted a special entitlement which gives the extension access to approximate eye input data. This eye input data may only be used to implement the foveated streaming protocol. - We've added a new API to FoveatedStreamingSession, queryStreamingProviders, which allows streaming clients to query the list of available protocols on the system. FoveatedStreamingSessions can select which provider to use via connect(endpoint:streamingProvider:)
- End-users install both the streaming protocol (MyProtocol.app, which vends MyProtocol.appex) as well as any games that leverage that protocol (MyGame.app, MyCADApplication.app, MySimulator.app). Those apps can choose to leverage MyProtocol.appex, when installed.
If you are interested in extending visionOS with a foveated streaming protocol, please reach out via our foveated streaming provider entitlement request form.
So to summarize, the FoveatedStreaming framework has two entitlements:
- Apps that use
FoveatedStreamingSessionneed the "Foveated Streaming Session" entitlementcom.apple.developer.foveated-streaming-session. This entitlement is available to any developers in the apple developer program with no review beyond App Store review. - App extensions that extend visionOS need the "Foveated Streaming Provider" entitlement
com.apple.developer.foveated-streaming-provider.- Only the streaming provider (MyProtocol.appex) needs this protected entitlement, because streaming providers are granted access to sensitive eye input data. Streaming client apps (e.g. MyGame.app) do not need this special
-providerentitlement, they only need-session.
- Only the streaming provider (MyProtocol.appex) needs this protected entitlement, because streaming providers are granted access to sensitive eye input data. Streaming client apps (e.g. MyGame.app) do not need this special
I also wanted to note how this architecture helps preserve user privacy:
- On visionOS, eye input data is restricted only to the app extension. App extensions are a separate process, and we have implemented technical restrictions to help ensure the eye input data does not leave the extension process.
- FoveatedStreamingProvider extension processes are only created when an end user begins a Streaming Session (ultimately via the
FoveatedStreamingSession.connectAPI). End users must give explicit permission every time they start this, and data is only sent for the duration of the session. Users may end the session (and therefore terminate the streaming provider extension) at any time, from Control Center. - The eye input data given to FoveatedStreamingProvider extensions is limited to the resolution required for high quality foveated streaming. Full resolution eye input data is not exposed.
- We review every application via the entitlement request form to help ensure that foveated streaming providers use eye input data responsibly.
