Revealed keyboard drifts in progressive immersion, and no immersion style gives windows-in-front + Crown + passthrough cutouts

Question

Created 4h

Replies 0

Boosts 0

Participants 1

Environment: Apple Vision Pro, visionOS 27.0.1 (24M372). Magic Keyboard, with Settings › Awareness & Safety › Reveal Keyboards set to Always.

Our app puts the user in an opaque virtual room while they work at their real desk, using app windows and Mac Virtual Display. A virtual desk is placed on the real desk. It is world-locked: positioned once from a PlaneAnchor (PlaneDetectionProvider), then a plain entity at fixed ImmersiveSpace coordinates. No AnchorEntity or WorldAnchor is involved. We offer two modes: .mixed, and .progressive for Digital Crown control. Each gives us only half of what desk work needs.

1. The revealed keyboard drifts in progressive immersion (FB25006789)

In .progressive, the physical keyboard shown by keyboard awareness does not stay registered with the virtual desk. After the keyboard has been out of view for a few seconds (for example, while looking at a Mac Virtual Display window), it looks "floaty" when you look back: as the head turns, it slides against the desk. Turning right, the keyboard moves left farther than the desk does.

What we checked:

It happens in .progressive at any Crown amount, and in .full. With identical content in .mixed (no app passthrough cutout, so again only keyboard awareness shows the keyboard), it is much steadier.
On 27.0.1, PlaneAnchor and DeviceAnchor coordinateSpace(correction: .rendered) are exactly equal to their .none transforms: 0 mm, 0°, scale 1. That held over 835 per-frame samples with ±50° yaw and up to 36 cm of head movement. So the rendered correction gives the app nothing to compensate with.
The app's desk transform is constant while the drift is visible.
upperLimbVisibility(.automatic) vs .visible and a .portrait progressive aspect ratio make no difference.

Possibly related: https://developer.apple.com/forums/thread/796557 (FB19666209).

2. No immersion style combines what desk work needs (FB25006837)

We need, at the same time:

windows (our own and Mac Virtual Display) never covered by the room's walls, furniture or trees;
the Digital Crown to bring the real surroundings back in;
a passthrough cutout over the real desk, and a well-registered keyboard.

.mixed:

✓ OcclusionMaterial cutouts reveal passthrough.
✓ The keyboard stays registered.
✗ RealityKit entities can render in front of windows, including Mac Virtual Display and window chrome. breakthroughEffect doesn't cover whole windows or Mac Virtual Display.
✗ The Crown can't adjust immersion.

.progressive:

✓ Windows always stay in front of immersive content (as an Apple engineer confirmed in https://developer.apple.com/forums/thread/764556).
✓ The Crown adjusts immersion.
✗ OcclusionMaterial reveals nothing inside the portal, so there is no desk cutout.
✗ The keyboard drifts (issue 1).

Because of issue 1 we have switched our default back to .mixed, and lost windows-in-front.

Questions

Is keyboard awareness in .progressive/.full expected to be registered less accurately than in .mixed? Is there a supported way to keep it registered with world-locked content, or to get the pose at which the system draws it?
In .mixed, is there a supported way to keep windows (including Mac Virtual Display) in front of an app's immersive content?
Or, in .progressive, a supported way to show passthrough in an app-defined region inside the portal?

Any pointers or workarounds are appreciated.