The goal is to achieve precise joint tracking for clinical assessment. The Doctor is wearing the AVP and observing the Patients movement. Do you have any recommended best practices for integrating real-time joint tracking and displaying them on the patient within visionOS? We attempted to use VNHumanBodyPose3DObservation, which theoretically should work, but we are unable to display the detected joints in an Immersive Space for real-time validation. This makes it difficult for the doctor to ensure accurate tracking and if possible a photo or video of the Range of Motion assessment would be needed for the patient record. Are there alternative methods to achieve precise real-time joint tracking without requiring main camera access (com.apple.developer.arkit.main-camera-access.allow)?

In ARKit for visionOS, I can track the user's head with a HeadAnchor, but it will not give the location. However, I can get the device's transform by calling queryDeviceAnchor(atTimestamp: CACurrentMediaTime()) on a WorldTrackingProvider. Why the difference? - if I know the device's transform, I effectively know the head's transform.

I need help to wrap my head around this... If I import the Reality Composer Pro package and load it into an ARView, I will see 1.3gb of memory usage and about 180-220% cpu usage. The frames will start at around 60fps, and then eventually drop to around 30fps. If I export the usdz from Reality Composer Pro and load that into the same ARView, I will see about 1gb of memory usage and around 150% cpu usage; fps holds longer at 60 but eventually drops. If I load that same usdz into a QuickLook view, I will see about 55mb of memory usage, 9-11% cpu, and the frames stay locked at 116fps. The only thing I notice is the button I have is slightly less responsive, but it all still works fine. I don't understand. How can I make the ARView work as efficiently as QuickLook?

When I've made an animated UDSZ, at what framerate will the animation be rendered in QuickLook? Is it the same across all devices? (iPhone, Apple Vision Pro, etc.) and viewing environments? (QuickLook, inside an ARView, etc.) Suppose I export my file at 30fps and the device draws at 60fps, does the device interpolate between frames automatically, animate at a lower frame rate, or play it at twice the speed? What if it were 24fps? My primary concern with understanding frame rates is a bit of trouble I've had making perfectly looping animations. There always seems to be the slightest stutter between iterations. Thanks in advance for any insights you're able to provide!

Hey I wanted to make an app that tracks changes in the room and room lightning and I was wondering if its possible to use VirtualEnvironmentProbeComponent to obtain the EnvironmentResource image and store it? If so are there any example of similar operation I could use? Thank you!

I have been referencing the Object Tracking Tutorial from WWDC 2024 on Vision OS, how Create ML is used to create a reference object, and we can track them in the ARSession. I am looking forward to building this feature on an AR app for iPhone, I am using iPhone 13 Pro Max. I have created couple of reference objects from the Create ML.

Hello, I am developing a visionOS application and am interested in obtaining detailed data of users’ hands through ARKit, including but not limited to Transform and rotation angle. I have reviewed Happy Beem, but it appears to only introduce the method of identifying the user’s specific gestures. Could you please advise on how to obtain the Transform and rotation angle of the user’s hand? Thank you.

We are using the ARKit image tracking feature on visionOS 2.0 with three pre-registered images. The image tracking works, but only one image is actively tracked at a time. When more than one target image is visible to the camera, it has difficulty detecting and tracking the other images. Is this the expected behavior in visionOS, or is there something we need to do to resolve this issue?

I am using ARKit to detect image in visionPro. However I met some question about adding the reference image. Some of my images can not be added correctly sometimes. (As you can see in the picture above, the 'orange' can not be added correctly, but the 'cup' can). However, sometimes they will be added without any problem. I do not know why it will happen. And I want they all be added steadily.

Hello. I am currently building an app using AR Kit. As for the UI, I am using SwiftUI and NavigationStack + NavigationLink for navigation and screen transitions! Here I need to go back and forth between the AR screen and other screens. If the number of screen transitions is small, this is not a problem. However, if the number of screen transitions increases to 10 or 20, it crashes somewhere. We are struggling with this problem. (The nature of the application requires multiple screen transitions.) The crash log showed the following. error: read memory from 0x1e387f2d4 failed AR_Crash_Sample-2025-03-07-115914.txt Incident Identifier: B23D806E-D578-4A95-8828-2A1E8D6BB7F8 Beta Identifier: 924A85AB-441C-41A7-9BC2-063940BDAF32 Hardware Model: iPhone16,1 Process: AR_Crash_Sample [2375] Path: /private/var/containers/Bundle/Application/FAC3D662-DB10-434E-A006-79B9515D8B7A/AR_Crash_Sample.app/AR_Crash_Sample Identifier: ar.crash.sample.AR.Crash.Sample Version: 1.0 (1) AppStoreTools: 16C7015 AppVariant: 1:iPhone16,1:18 Beta: YES Code Type: ARM-64 (Native) Role: Foreground Parent Process: launchd [1] Coalition: ar.crash.sample.AR.Crash.Sample [1464] Date/Time: 2025-03-07 11:59:14.3691 +0900 Launch Time: 2025-03-07 11:57:47.3955 +0900 OS Version: iPhone OS 18.3.1 (22D72) Release Type: User Baseband Version: 2.40.05 Report Version: 104 Exception Type: EXC_CRASH (SIGABRT) Exception Codes: 0x0000000000000000, 0x0000000000000000 Termination Reason: SIGNAL 6 Abort trap: 6 Terminating Process: AR_Crash_Sample [2375] Triggered by Thread: 7 Application Specific Information: abort() called Thread 7 name: Dispatch queue: com.apple.arkit.depthtechnique Thread 7 Crashed: 0 libsystem_kernel.dylib 0x1e387f2d4 __pthread_kill + 8 1 libsystem_pthread.dylib 0x21cedd59c pthread_kill + 268 2 libsystem_c.dylib 0x199f98b08 abort + 128 3 libc++abi.dylib 0x21ce035b8 abort_message + 132 4 libc++abi.dylib 0x21cdf1b90 demangling_terminate_handler() + 320 5 libobjc.A.dylib 0x18f6c72d4 _objc_terminate() + 172 6 libc++abi.dylib 0x21ce0287c std::__terminate(void (*)()) + 16 7 libc++abi.dylib 0x21ce02820 std::terminate() + 108 8 libdispatch.dylib 0x199edefbc _dispatch_client_callout + 40 9 libdispatch.dylib 0x199ee65cc _dispatch_lane_serial_drain + 768 10 libdispatch.dylib 0x199ee7158 _dispatch_lane_invoke + 432 11 libdispatch.dylib 0x199ee85c0 _dispatch_workloop_invoke + 1744 12 libdispatch.dylib 0x199ef238c _dispatch_root_queue_drain_deferred_wlh + 288 13 libdispatch.dylib 0x199ef1bd8 _dispatch_workloop_worker_thread + 540 14 libsystem_pthread.dylib 0x21ced8680 _pthread_wqthread + 288 15 libsystem_pthread.dylib 0x21ced6474 start_wqthread + 8 Perhaps I am using too much memory! How can I address this phenomenon? For the AR functionality, we are using UIViewRepresentable, which is written in UIKit and can be called from SwiftUI import ARKit import AsyncAlgorithms import AVFoundation import SCNLine import SwiftUI internal struct MeasureARViewContainer: UIViewRepresentable { @Binding var tapCount: Int @Binding var distance: Double? @Binding var currentIndex: Int var focusSquare: FocusSquare = FocusSquare() let coachingOverlay: ARCoachingOverlayView = ARCoachingOverlayView() func makeUIView(context: Context) -> ARSCNView { let arView: ARSCNView = ARSCNView() arView.delegate = context.coordinator let configuration: ARWorldTrackingConfiguration = ARWorldTrackingConfiguration() configuration.planeDetection = [.horizontal, .vertical] if ARWorldTrackingConfiguration.supportsFrameSemantics(.sceneDepth) { configuration.frameSemantics = [.sceneDepth, .smoothedSceneDepth] } arView.session.run(configuration, options: [.resetTracking, .removeExistingAnchors]) context.coordinator.sceneView = arView context.coordinator.scanTarget() coachingOverlay.session = arView.session coachingOverlay.delegate = context.coordinator coachingOverlay.goal = .horizontalPlane coachingOverlay.activatesAutomatically = true coachingOverlay.autoresizingMask = [.flexibleWidth, .flexibleHeight] coachingOverlay.translatesAutoresizingMaskIntoConstraints = false arView.addSubview(coachingOverlay) return arView } func updateUIView(_ _: ARSCNView, context: Context) { context.coordinator.mode = MeasurementMode(rawValue: currentIndex) ?? .width if tapCount == 0 { context.coordinator.resetMeasurement() return } if distance != nil { return } DispatchQueue.main.async { if context.coordinator.distance == nil { context.coordinator.handleTap() } } } static func dismantleUIView(_ uiView: ARSCNView, coordinator: Coordinator) { uiView.session.pause() coordinator.stopScanTarget() coordinator.stopSpeech() DispatchQueue.main.async { uiView.removeFromSuperview() } } func makeCoordinator() -> Coordinator { Coordinator(self) } class Coordinator: NSObject, ARSCNViewDelegate, ARSessionDelegate, ARCoachingOverlayViewDelegate { var parent: MeasureARViewContainer var sceneView: ARSCNView? var startPosition: SCNVector3? var pointedCount: Int = 0 var distance: Float? var mode: MeasurementMode = .width let synthesizer: AVSpeechSynthesizer = AVSpeechSynthesizer() var scanTargetTask: Task<Void, Never>? var currentResult: ARRaycastResult? init(_ parent: MeasureARViewContainer) { self.parent = parent } // ... etc } }

I would like to know if there have been any updates that improved the image tracking speed. I submitted feedback a while ago, mentioning that the image tracking rate was only one frame per second on Vision Pro. Thank you very much. FB15688516 and FB16745373is feedback we previously submitted, but we have not received any response yet.

As I understand it there are two ways I can track a hand, or a joint, in RealityKit: either, create an AnchorEntity, for example AnchorEntity(.hand(.left, location: .palm)) or, set up an ARSession with a HandTrackingProvider ( a lot more code which I haven't repeated here). Assuming this is correct, when would I want to use one over the other?

My app sends users notifications of important messages in the app. How can I make sure the user receives them?

We're developing a VisionOS application, where we would like to do product recognition (like food items). We have enterprise entitlements and therefore also main camera access for VisionOS. We send this live camera frames to a trained CoreML model where we will receive 2D coordinates from the model detection prediction. Now, we would like to create a 3D anchor on the detected items so it can be visible for user. The 3D anchor is going to be the class name of the detected item. How do we transform this 2D coordinate from the model prediction to a 3D anchor?

I am working with MeshAnchors, and I am having troubles getting to the classification of the triangles/faces. This post references the MeshAnchor.Geometry, and that struct does have a property named "classifications", but it is of type GeometrySource. I cannot find any classification information in GeometrySource. Am I missing something there? I think I am looking for something of type MeshAnchor.MeshClassification, but I cannot find any structs with this as a property.

I am developing an visionos app. I load a .usdz file as a Reality Entity(such as a cabbage). And I want such an effect: When I turn on a desk lamp in real world near the Entity, the surface of the Entity will correctly respond to the light in the real world. I want an effect like this: https://www.reddit.com/r/virtualreality/comments/1as01mm/shiny_disco_ball_reflecting_my_room/ I look up the api such as ImageBasedLightComponent andVirtualEnvironmentProbeComponent in RealityKit、EnvironmentLightEstimationProvider in ARKit，but I do not know how to code. Besides, it will be better if the shadow will also respond to the light correctly.

I am developing an ARKit based application that requires plane detection of the tabletop at which the user is seated. Early testing was with an iPhone 8 and iPhone 8+. With those devices, ARKit rapidly detected the plane of the tabletop when it was only 8 to 10 inches away. Using iPhone 15 with the same code, it seems to require me to move the phone more like 15 to 16 inches away before detecting the plane of the table. This is an awkward motion for a user seated at a table. To validate that it was not necessarily a feature of my code, I determined that the same behavior results with Apple's sample AR Interaction application. Has anyone else experienced this, and if so, have suggestions to improve the situation?

Hi, is there any way to use front camera to do the motion capture? I want recognize if the user raised there hands up with the front camera on iPhone. I was able to do it with the back camera, not the front. Also, if there is any sample code, or document, I would be super happy. Waiting for your reply!!

Hello Everyone Sam Francisco here otherwise known as Ian Francis Creative Director at Primus Films Ltd in the UK. I'm excited today to have Apple invite me to the Developer Forums and indeed honoured to have the chance of creating something that is of interest in the App Store and beyond. My question is self explanatory really, I am in the Final Cut of 'Metaphor' which is a 42 minutes 17' second Horror Film about Climate Change and I have a deadline set for it's Premiere on YouTube which is this year on April 22nd which as I'm sure you all know is Earth Day. So although I am quite comfortable and relatively proficient as the film's Writer, Director and Editor I have to confess to being green as tomatoes when it comes to marketing and publicity, and negotiating rights and agreeing deals for example to gain access to the App Store here at Apple with a view to discussing rental or sale options. Even if the film comes away empty handed from the Dorothy Chandler it'll be all set for the Baftas over in the UK and I was wondering how to get discussions going on this and if anyone out there has any knowledge of where I even start I would be very grateful. Thank you for your time Ian Francis

Hi, we would like to create something where you can open multiple volumetric windows and place them in a room, our biggest issue is that we want these windows to be persistent, so when I close and reopen the app, the windows to be in the same position. We can't use immersive spaces because we also want to have the possibility to access the shared space. Is it possible with the current features and capabilities to do that? If yes do you have some advices how can we achieve this? The alternative is if is it possible to open the virtual display in immersive spaces or if we have the possibility to implement our own virtual display.