Hi,
We're building a mixed-immersive visionOS app in which the user holds a thin metal
tool — a needle — near virtual content. We need the real needle to correctly
occlude virtual content: when the needle (and especially its tip) is physically in
front of a virtual object, the virtual object must not draw over it.
We are solution-agnostic. Whether the right answer is LiDAR / scene-depth based,
object-tracking based, hand-tracking based, or something else entirely — we just need
needle occlusion to work. Here is what we've ruled out so far:
ObjectTrackingProvider — blocked at enrolment. We can't even produce a usable
reference object: the object-capture / scanning step doesn't pick up the needle at
all (≈1 mm shaft, specular/metallic, almost no surface features). So object
tracking never gets off the ground — it fails before runtime.
SceneReconstructionProvider + OcclusionMaterial — only usable for large planar
surfaces. Occlusion against a wall or floor is clean. But the mesh is too coarse for
anything with finer geometry: a chair already gives very rough, ragged occlusion
edges, and the needle is far below the mesh resolution — its shaft never appears in
the mesh, so it never occludes at all. So this isn't only a needle problem: it's a mesh
resolution problem that goes from rough (chair) to nonexistent (needle).
Automatic hand occlusion — doesn't cover the tip. The hand is composited in
front correctly, but the needle tip protrudes a few centimetres beyond the fingers,
and virtual content draws over exactly that part.
The needle tip is the part that most needs to occlude correctly, so this is blocking
for us.
The full Xcode sample project and the screen recording are in this GitHub repo:
https://github.com/JerryNee/occlusion-demo (the video is at media/occlusion-demo.mp4).
The screenshots below are stills from that recording, which has four stages, all with the
same virtual cube:
Occlusion off — baseline; the cube draws over everything in the room.
Occlusion on, against a wall — the cube is occluded cleanly.
Occlusion on, against a chair — the cube is occluded, but with rough, ragged edges.
Occlusion on, with a needle — the needle does not occlude the cube at all.
The same occlusion path therefore degrades from clean (wall) → rough (chair) →
nonexistent (needle) as the real geometry gets finer.
Questions / avenues we'd welcome guidance on (any one would unblock us):
What is the recommended way to occlude an object this thin on visionOS — and is
it possible at all today?
Is there app-accessible per-pixel scene depth (analogous to ARFrame.sceneDepth
on iOS) that we could use to occlude geometry the reconstruction mesh misses?
Can the system's automatic hand / upper-limb occlusion be extended to include a
thin instrument held in the hand — i.e. treat a held tool as part of the hand region?
The tool has known, rigid geometry, so we considered attaching an
OcclusionMaterial proxy mesh and positioning it from hand-tracking joints —
but the needle's orientation within the grip is not determined by the hand pose
(the same grip can hold the needle at many different angles), so a hand-joint proxy
would be misaligned. Is there any supported way to recover the actual 6-DoF pose
of a thin tool held in the hand (i.e. something that senses the tool itself,
since the hand pose doesn't constrain it)?
Is there any way to make object capture / scanning succeed for thin, specular
objects, or an alternative enrolment path?
Environment: Apple Vision Pro, visionOS 26.2.
Full Xcode sample + screen recording: https://github.com/JerryNee/occlusion-demo
Even a "not currently possible / known limitation" answer would help us plan our
roadmap. Thank you!
2
0
418