Dear all, We are building an XR application demonstrating our research on open-vocabulary 3D instance segmentation for assistive technology. We intend on bringing it to visionOS using the new Enterprise APIs. Our method was trained on datasets resembling ScanNet which contain the following: localized (1) RGB camera frames (2) with Depth (3) and camera intrinsics (4) point cloud (5) I understand, we can query (1), (2), and (4) from the CameraFrameProvider. As for (3) and (4), it is unclear to me if/how we can obtain that data. In handheld ARKit, this example project demos how the depthMap can be used to simulate raw point clouds. However, this property doesn't seem to be available in visionOS. Is there some way for us to obtain depth data associated with camera frames? "Faking" depth data from the SceneReconstructionProvider-generated meshes is too coarse for our method. I hope I'm just missing some detail and there's some way to configure CameraFrameProvider to also deliver depth and/or point clouds. Thanks for any help or pointer in the right direction! ~ Alex

I"m trying to create a simple app for my students that will display .heic images taken with a nikon and them converted to .heic in the photos app. My attempts only result in the QuickLook viewer showing the images in 2d. Any guidance? Here is my ContentView: import SwiftUI import QuickLook struct ContentView: View { @State private var showQuickLook = false @State private var previewURL: URL? = nil // State to store the URL for Quick Look var body: some View { VStack { Button("See it in 3D") { // Set the URL for the file from the bundle and toggle Quick Look presentation if let imageURL = Bundle.main.url(forResource: "Michelia_fuego", withExtension: "heic") { previewURL = imageURL // Set the preview URL if the image is found showQuickLook.toggle() // Toggle to trigger Quick Look presentation } else { print("File not found") // Print error if the file is missing } } .quickLookPreview($previewURL) // Binding to the URL } } } #Preview { ContentView() }

As far as I know, Apple hasn’t opened access to the Vision Pro camera for developers yet, so I’m trying to find possible workarounds within the current capabilities. I’m wondering if there’s any way to apply a mesh to a person in the scene in Vision Pro, or if there’s an alternative approach to roughly detect a human shape in front of the user?

Hi, I'm working with CameraFrameProvider from Enterprise API. Is it always capped at 30fps, or is there something I can switch to get more? I assume it is capped at 30, so let me cram in additional question here :). If I'd get a developer strap and attach an external camera capable of doing >30fps, will I get the full stream, or some other limitation will kick in?

I want to make a model with added bones move by dragging it with gestures，This is a model exported using Blender， What I understand is using IKComponent, but I don't know how to use it specifically

When building a multiplayer Tabletop game, the documentation includes how to attach a custom TabletopNetworkSessionCoordinator, which could be used in addition to TabletopGame.MultiplayerDelegate. But so far, we have been unable to create these types of custom coordinators or have a delegate that works. Our current setup with our generic GroupActivity works by sending the session to TabletopGame's coordinateWithSession method (like in the current sample project), but we didn't find a way to access and control, for example, the arbiter, seats, player events, among other features mentioned on https://developer.apple.com/documentation/tabletopkit/tabletopnetworksession. Is correct to expect having access to the participants, messenger, or journal without having to maintain a parallel coordinator?   possibly we are missing something here; any suggestions?

I am looking for a material that functions in the same way that Occlusion Material does, except that it only partially occludes whatever is behind it. One way that I have thought of doing this was to change the opacity of the entity that was covered in Occlusion Material, however this did not change anything. Please let me know if this is possible.

Is it possible to detect distance from the vision pro to real live objects and people? I tried using scene.raycast to perform a raycast forward from the center of the viewport, but it doesn't seem to react to real life objects, only entities. I see mentioned here: https://developer.apple.com/forums/thread/776807?answerId=829576022#829576022, that a raycast with scene reconstruction should allow me to measure that distance, as long as the object is non-moving. How could I accomplish that?

From visionOS 2.0 we can access Apple Vision Pro's main camera but only for Enterprise account as it is enterprise API only, I have a normal Developer account and I want to use main camera and want to have a video call feature in app by using main camera of AVP, is it possible to do it using developer account only. Currently using that account I am not able to create entitlement certificate as there is no option.

I have a simple example of a motion matching (MxM for Unity) character controller that uses Unity's input system and gamepad support. In editor the scene and inputs work as expected. When I build to headset the app stops at an initialization step where my game controller should kick in. The app doesn't crash but my character is frozen in A-Pose and doesn't respond to input. I'm wondering if this error I'm seeing in the logs is what's causing it? And if so how do I fix it? error 15:56:11.724200-0700 PolySpatialProjectTemplate NSBundle file:///System/Library/Frameworks/GameController.framework/ principal class is nil because all fallbacks have failed I'm using Xcode 16 beta 6 Unity 6000.0.17f1 VisionOS 2.0 beta 9

Hello, Is there a way to always have the attachments of a RealityView always face the user? For example, in a visionOS app, in an immersive space, we have an attachment. When the user either walks around the attachment, or rotates the parent entity, we would like the attachment to automatically rotate to face the user. How do we do this? I anticipated this to be a trivial feature to implement, since I thought I remembered seeing this feature as a built-in/opt-in option for attachments. But, I cannot find that feature. All and any recommendations are appreciated, thanks.

What is recommended best practice for importing a Blender 3D file into RCP? I assume as a .usdz file? Is there a WWDC24 session or other Apple resource that best explains this. I want to make sure I provide the right format/file to RCP from Blender.

Hello everyone, I've been trying for a few weeks now to convert a sequential series of meshes into a stop-motion animation in USDZ format. In Unreal Engine, I’ve already figured out how to transform the sequential series of individual meshes into a smooth animation using the node system and arrays. Unfortunately, the node system cannot be exported as a usdz animation logic in either Unreal or Blender. Because of this, I have tried several other methods to incorporate the animation logic. Here’s what I’ve tried so far: I attempted to create the animation in Blender with Render-/Viewports and mapping it to keyframes. However, in my experience, Viewports are not supported in the conversion. I tried aligning the vertices of individual objects and merging the frames using the Shrinkwrap modifier in Blender, then setting up a morph animation with keyframes. However, because the individual meshes are too different, this results in artifacts, and manually editing each mesh is too difficult for me to handle. I placed all individual meshes at the same position and animated them sequentially by scaling them from 0 to 100 in keyframes (Frame 1 is visible for 10 frames, then scales down at frame 11, while Frame 2 becomes visible at frame 11, and so on). I also adjusted the keyframes so that the scaling happens in a "constant" manner rather than the default Bezier or linear interpolation. I then converted this animation to .abc, and the result initially looked good. However, some information is lost when converting it with OpenUSD. The animation does not maintain its intended jump-like behavior in USDZ format, and instead, the scaling of individual files is visible in the animation. I tried using a Blender add-on (StepMotion), which allows the animation to be exported as .abc, but it can only be read in Blender or Unreal. Even in the preview, the animation is not displayed correctly, so converting the animation logic does not work either. 
Unfortunately, I have no alternative way to create the animation, as the individual frames have been provided to me as meshes. So far, I haven’t found a way to implement this successfully. I would be very grateful for any tips or ideas, as I am running out of options on how to make this work. Thanks in advance!

Hi! I'm trying to play video on monitor 3D model like a material. I wanna know if it's possible work. I searched about it, but I couldn't get enough information. Thank you in advance.

I have created a portal and attached it to a wall using the AnchorEntity. However, I am seeking guidance on how to determine the size of the wall so that the portal can fully occupy it. Initially, I attempted to locate relevant information within the demo code, but I encountered difficulties in comprehending certain sections. I would appreciate it if someone could provide a step-by-step explanation or a reference to the appropriate code. Thank you for your assistance.

In visionOS beta, when using ARKit for image detection, the initially detected AnchorUpdate status is .add, and subsequent detections of the same image are marked as .update. However, after toggling immersiveSpace, the same image is detected with the status .add again. After updating to visionOS 2.1, the first detection status remains `add, and subsequent detections of the same image remain .update, even after toggling immersiveSpace. Could this be due to a change in processing flow?

We have a project which is currently being built as a XCFramework. The framework contains a custom component to be used with entities in Reality Composer Pro. I have tried to se set the RCP Package.swift file to reference the framework package for the in the dependancies. Nothing that I do with the folder path to reference the code is working. Do I need to change the project to be using Swift source code instead of a XCFramework? The component needs to be in the framework as there is a class in the framework that works directly with the custom compoent. I am able to reference the XCFramework as a Swift Package with other projects.

"Although Xcode generates loading methods for all Reality Composer files in your Xcode project" I do not find this to be true, sadly. Does anyone have any luck or insight on how one can build just a simple MacOS app that will import a scene from a Reality File? The documentation suggests that the simple act of bringing a .Reality File in (What about .realitycomposerpro?) will generate code, but that doesn't seem to happen. The sample code (Spaceship) does not compile for MacOS. I'd really love just the most generic template of an Xcode Project that compiles with a button that pops open a scene., Like the VisionOS default immersive project.

The setup:
 My Vision Pro app loads uszd models created by a 3rd party app. These models have to be scaled to the right dimension for RealityKit. I want to use physics for realistic movement of these entities, but this does not work as expected. I thus wrote a demo test app based on Apple's immersive space app template (code below).
This app has two entities, a board and above a box that is a child of the board.
 Both have a collision component and a physics body.
 The board, and thus also its child the box, are scaled. The problem:
 If the scale factor is greater or equal 0.91, the box falls under gravity towards the board where it is stopped after some movements. This looks realistic.
 However, if the scale factor is below 0.91, even 0.9, the movement of the box is immediately stopped on the board without any physics simulation.
 Unfortunately, I am not able to upload screen recordings, but if the demo app is executed, one sees the effect immediately. The question:
 I cannot imagine that this behavior is correct. Can somebody confirm that this is a bug? If so I will write a bug report.
 My demo app uses Xcode Version 16.0 (16A242d), simulator Version 16.0 (1038) and visionOS 2.0. The code: import SwiftUI import RealityKit import RealityKitContent struct ImmersiveView: View { @Environment(AppModel.self) var appModel var body: some View { RealityView { content in let boardEntity = makeBoard() content.add(boardEntity) let boxEntity = makeBox() boardEntity.addChild(boxEntity) moveBox(boxEntity, parentEntity: boardEntity) } } func makeBoard() -> ModelEntity { let mesh = MeshResource.generateBox(width: 1.0, height: 0.05, depth: 1.0) var material = UnlitMaterial(); material.color.tint = .red let boardEntity = ModelEntity(mesh: mesh, materials: [material]) let scale: Float = 0.91 // Physics does not run if scale < 0.91 boardEntity.scale = [scale, scale, scale] boardEntity.transform.translation = [0, 0, -3] boardEntity.generateCollisionShapes(recursive: false) boardEntity.physicsBody = PhysicsBodyComponent(massProperties: .default, material: PhysicsMaterialResource.generate(friction: .infinity, restitution: 0.8), mode: .static) return boardEntity } func makeBox() -> ModelEntity { let mesh = MeshResource.generateBox(size: 0.2) var material = UnlitMaterial(); material.color.tint = .green let boxEntity = ModelEntity(mesh: mesh, materials: [material]) boxEntity.generateCollisionShapes(recursive: false) boxEntity.physicsBody = PhysicsBodyComponent(massProperties: .default, material: PhysicsMaterialResource.generate(friction: .infinity, restitution: 0.8), mode: .dynamic) return boxEntity } func moveBox(_ boxEntity:Entity, parentEntity: Entity) { // Set position and orientation of the box let translation = SIMD3<Float>(0, 0.5, 0) // Turn the box by 45 degrees around the y axis let rotationY = simd_quatf(angle: Float(45.0 * .pi/180.0), axis: SIMD3(x: 0, y: 1, z: 0)) let transform = Transform(rotation: rotationY, translation: translation) boxEntity.transform = transform } }

I tried to use the application icon from sample project https://developer.apple.com/documentation/visionos/diorama, but the 3 layers of the app icon are not separated when I hover on the icon in the Vision Pro simulator. Could you please advise how to fix the problem? I am using the latest Xcode Version 15.4 (15F31d). Thank you.