Is it possible to access the raw lidar measurements before the sceneDepth calculation is done to combines the lidar measurements with visual data. In low light environments the lidar scanner should still work and provide depth info but I cannot figure out how to access those pure lidar depth measurements. I am currently using: guard let frame = arView.session.currentFrame, let depthData = frame.sceneDepth?.depthMap else { print("Depth data is unavailable.") return } but this is the depth data after sensor fusion occurs and fails in low light conditions.

I created an app where the user: first scans a room using RoomCaptureView (RoomPlan) then taps on physical elements (objects, walls...) using an ARView to record some 3d positions I can handle taps in an ARView using a UITapGestureRecognizer and the ARView raycast(from:, allowing:, alignment:) method. This works fine, so I thought I could do the same using the ARView used by RoomCaptureView., so the user can scan a room and record some 3d positions at the same time. Sadly, this approach does not work, as the raycast method always returns nil. What I actually need is mapping a tap on screen to a real-world position during RoomCaptureSession. Does anyone know how to do this?

After creating a ". plist" file in the "Document" folder of the app, uninstalling and reinstalling the app, the "FileManager. default. fileExists (atPath: folderURL. path())" code returns true. I checked if the ". plist" file was not found in the "Document" folder of the app. Is this a bug in the VisionOS system

How do we author a Reality File like the ones under Examples with animations at https://developer.apple.com/augmented-reality/quick-look/ ?? For example, "The Hab" : https://developer.apple.com/augmented-reality/quick-look/models/hab/hab_en.reality Tapping on various buttons in this experience triggers various complex animations. I don't see any way to accomplish this in Reality Composer. And I don't see any way to export/compile to a "reality file" from within Xcode. How can I use multiple animations within a single GLTF file? How can I set up multiple "tap target" on a single object, where each one triggers a different action? How do we author something similar? What tools do we use? Thanks

Lately i got a lot of crashes on iOS 18 devices (mostly 13 Pro devices but also a 16 Pro Max), has anyone encountered similar issues and is there more information about offlineFloorPlanGeneration? com.apple.RoomScanCore.offlineFloorPlanGeneration EXC_BAD_ACCESS KERN_INVALID_ADDRESS 0x0000f5cab18e03c0 RoomScanCore RSFrameFromDictionary + 110992

I am attempting to use the Barcode Detection enterprise API. I have the necessary entitlements and license file. I'm following the sample code online, and whenever I attempt to run the barcode detection using arKitSession.run I get the following error message: ar_barcode_detection_provider_t <0x300d82130>: Failed to run provider with transient error code: 1 It obviously isn't running the barcode detection, even though it's running in an immersive space in mixed mode. Any idea what might be going on?

Hello, i've recently received the entitlements to access the main camera stream for a project on the Apple Vision Pro. What happens : When executing code from this WWDC tutorial , i'm getting this error when trying to use a Camera Frame Provider : ar_camera_frame_provider_t <0x300d58870>: Failed to start camera stream with error: <ar_error_t: 0x303fcc4c0 Error Domain=com.apple.arkit Code=100 "App not authorized." UserInfo={NSLocalizedFailureReason=Using camera frame provider requires an entitlement., NSLocalizedRecoverySuggestion=, NSLocalizedDescription=App not authorized.} What I've tried : I followed the instructions given by mail, by : adding the .license file at the root of my project, adding the .entitlements file by adding capabilities in the project (Main Camera Access & Passthrough in screen capture are there). I've added NSCameraDescription, NSEnterpriseMCAMUsageDescription and NSWorldSensingUsageDescription (they all have a value assigned). I've also followed those post & post advices. When checking on the Account settings, i do see the capabilities in the "additional capabilities" On first launch, I'm also getting prompted to accept the NSEnterpriseMCAMUsageDescription, so I assume the info.plist file is valid? What did i missed to get the entitlements working ? Here's the code : import ARKit import SwiftUI import Vision import RealityKit class MainCameraAccess { var arKitSession = ARKitSession() var cameraFrameProvider = CameraFrameProvider() var pixelBuffer: CVPixelBuffer? func startCameraSession() async { let formats = CameraVideoFormat.supportedVideoFormats(for: .main, cameraPositions: [.left]) // Request authorization await arKitSession.requestAuthorization(for: [.cameraAccess]) // Start the session do { try await arKitSession.run([cameraFrameProvider]) } catch { print("Failed to start ARKit session: \(error)") return } // Get camera frame updates guard let cameraFrameUpdates = cameraFrameProvider.cameraFrameUpdates(for: formats[0]) else { return } // Process frames for await cameraFrame in cameraFrameUpdates { guard let mainCameraSample = cameraFrame.sample(for: .left) else { continue } self.pixelBuffer = mainCameraSample.pixelBuffer } } func saveLatestImage() { guard let pixelBuffer = self.pixelBuffer else { print("No image available to save.") return } // Convert CVPixelBuffer to UIImage let ciImage = CIImage(cvPixelBuffer: pixelBuffer) let context = CIContext() guard let cgImage = context.createCGImage(ciImage, from: ciImage.extent) else { print("Failed to create CGImage.") return } let uiImage = UIImage(cgImage: cgImage) // Save UIImage to Photos Album UIImageWriteToSavedPhotosAlbum(uiImage, nil, nil, nil) print("Image saved to photo library.") } } Thanks in advance for the help, Jeremy

Hi, in visionOS2, I wonder if there is a method to add/remove referenceObjects of ObjectTrackingProvider after ARSession started without stop it. Now I must stop it and run it again. The current tracking is stopped. Thanks.

Hi everyone, I’m having trouble with image anchoring when working on a project in Reality Composer and Reality Composer Pro. Here’s the issue: 1. What I’m Trying to Achieve: I want to create an AR scene where an object anchors to an image I provide. I don't want to create an app for this but just use the USDZ File the Scene creates. The USDZ File then should be viewable via the various integrations of AR Quick Look across the Apple Ecosystem. The image anchoring works perfectly when I preview the scene inside Reality Composer using AR mode. 2. The Problem: When I export the project (tried both USDZ and Reality formats) and open it on my iPhone using the Files app (which uses AR Quick Look), the image anchoring no longer works. The object doesn’t anchor to the provided image as expected. It just anchors to the first plane it recognizes and not the image. 3. What I’ve Tried: Exporting the scene in USDZ format. Exporting the scene in Reality format. Both formats result in the same issue: no image anchoring outside of the Reality Composer environment. Trying different images but all resulting in same manor that the image anchoring is not working Tried different iOS Version but resulting in the same issue 4. Current Setup: Reality Composer Pro version: 2.0 iPhone model: iPhone 13 Pro iOS version: 18.1. 5. What I Need Help With: Is there a way to ensure image anchoring works in exported files when opened via AR Quick Look? Do I need to configure something specific during the export process? Are there limitations in AR Quick Look that prevent image anchoring from functioning correctly? Do i need to create an app to make this work? I’d appreciate any advice or insights from the community. If anyone has experience with similar issues or knows of a workaround, please let me know! Thanks in advance, Mav

Hi, I added DockingRegion to my scene from Reality Composer Pro, and I am able to load up the scene, but DockingRegion is getting ignored and the scene is getting rendered with no change in AVPlayerViewController window. As it can be seen in Reality Composer Pro screenshot below, I set the width of the player to 666, and moved it to the back by 300cm, but the actual result does not reflect the position I set on Reality Composer Pro. Is there anything else I should do other than loading up the Entity and adding to RealityView? Specifically, do I have to get the DockingRegion within the usda file and somehow enable it?

I’d like to use ARKit world tracking and display both the back camera feed and the front camera feeds, using the front feed as as a PIP. This would work great for an internet streaming use case. However, it’s impossible. As soon as ARKit is told to use one mode, the camera for the other side freezes/doesn’t work. This page also says you have to pick one camera to show: https://developer.apple.com/documentation/arkit/arkit_in_ios/choosing_which_camera_feed_to_augment?language=objc A question to the developers: why is this limitation in-place? Are there any work-arounds for the use case of ARKit world tracking + displaying the back camera feed + displaying the front camera feed as an overlay? It’s possible to do this with plain camera initialization without ARKit. (There’s an official example.) With ARKit, it no longer works. It’s strange that I cannot access the front feed via one of the other frameworks, but I guess that ARKit blocks that.

I have a VideoMaterial inside a RealityView and want to attach this to a DockingRegion inside an immersive environment. It appears that adding the VideoMaterial entity as a child of the docking region somewhat works, but there are no lighting effects (specular, diffuse) from the playing video. So essentially, how can you add a VideoMaterial to a DockingRegion and achieve the same reflections/behavior as using AVPlayerViewController. The latter is not an option as I need custom controls.

We have a plane model (basecircle)without physics and rigid body components, and no gestures are implemented. However, when tapped, the model unexpectedly falls into infinity. func fetchEnvResource(){ var simpleMaterial = SimpleMaterial() env = try! Entity.loadModel(named: "bgMain5") env.position += [0,0,-10] envTexture = PhysicallyBasedMaterial() envTextureUnlit = UnlitMaterial() envTexture.baseColor = .init(texture: .init(try! .load(named: "bgMain5"))) envTextureUnlit.color = .init(texture: .init(try! .load(named: "bgMain5"))) env.isEnabled = false let anchor = AnchorEntity(world: [0, 0, -3]) baseCircle = ModelEntity(mesh: .generatePlane(width: 1.5, depth: 1.5, cornerRadius: 0.75), materials: [SimpleMaterial(color: .green, isMetallic: false)]) env.components.set(InputTargetComponent()) baseMaterial = PhysicallyBasedMaterial() baseMaterialUnlit = UnlitMaterial() baseMaterial.baseColor = .init(texture: .init(try! .load(named: "groundTexture"))) baseMaterial.baseColor.tint = UIColor(white: 1.0, alpha: CGFloat(textureOpacity)) baseCircle.model?.materials = [baseMaterial] baseCircle.generateCollisionShapes(recursive: false) baseCircle.components.set(InputTargetComponent()) baseCircle.components[PhysicsBodyComponent.self] = .init(PhysicsBodyComponent(massProperties: .default, mode: .static)) baseCircle.physicsBody = PhysicsBodyComponent( mode: .kinematic ) anchorEntity.addChild(baseCircle) baseCircle.position = [0,0,-3] baseCircle.isEnabled = false let cylinder = ModelEntity(mesh: .generateCylinder(height: 0.2, radius: 0.5), materials: [SimpleMaterial(color: .blue, isMetallic: false)]) cylinder.position = [0,-0.1,0] cylinder.generateCollisionShapes(recursive: false) cylinder.components[PhysicsBodyComponent.self] = .init(PhysicsBodyComponent(massProperties: .default, mode: .static)) cylinder.physicsBody = nil cylinder.scale = [500, 100, 100] anchor.addChild(cylinder) } The plane model in this issue is the BaseCircle. Any suggestions on how to solve this or potential fixes would be greatly appreciated

Hey, I have Enterprise Access on the account and have added the passthrough capability and the entitlement on the main project and the "Broadcast Upload" extension, too. The broadcast works except it returns a black screen. I am attaching some screenshots below of the entitlement file. I have tried searching online to no avail, so any help would be greatly appreciated. I am also attaching the code. import Foundation import AVFoundation import ReplayKit class VideoAssetWriter { private var isRecording = false private var outputStream: OutputStream? private func setupConnection() { guard outputStream == nil else { return } print("setting up connection.") let serverIP = macIP let port = 12345 var readStream: Unmanaged<CFReadStream>? var writeStream: Unmanaged<CFWriteStream>? CFStreamCreatePairWithSocketToHost(kCFAllocatorDefault, serverIP as CFString, UInt32(port), &readStream, &writeStream) guard let writeStream = writeStream?.takeRetainedValue() else { print("Failed to create write stream") return } self.outputStream = writeStream as OutputStream self.outputStream?.open() } func startRecording() { isRecording = true } func processVideoSampleBuffer(_ sampleBuffer: CMSampleBuffer) { print("Processing Sample 1") guard isRecording else { return } print("Processing Sample 2") sendVideoChunkToServer(sampleBuffer) } private func sendVideoChunkToServer(_ sampleBuffer: CMSampleBuffer) { guard let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return } print("Processing Sample 3") let ciImage = CIImage(cvPixelBuffer: imageBuffer) let context = CIContext() guard let cgImage = context.createCGImage(ciImage, from: ciImage.extent) else { return } print("Processing Sample 4") let image = UIImage(cgImage: cgImage) if let imageData = image.jpegData(compressionQuality: 0.5) { guard imageData.count <= 10_000_000 else { print("Frame too large: \(imageData.count) bytes") return } if outputStream == nil { setupConnection() } print("sending frame size up connection.") // Convert to network byte order (big-endian) var frameSize = UInt32(imageData.count).bigEndian let sizeData = Data(bytes: &frameSize, count: MemoryLayout<UInt32>.size) _ = sizeData.withUnsafeBytes { outputStream?.write($0.baseAddress!.assumingMemoryBound(to: UInt8.self), maxLength: sizeData.count) } print("sending image data up connection.") // Send frame data _ = imageData.withUnsafeBytes { outputStream?.write($0.baseAddress!.assumingMemoryBound(to: UInt8.self), maxLength: imageData.count) } } } func stopRecording() { isRecording = false outputStream?.close() outputStream = nil } } This is the broadcast picker view wrapper: // Broadcast Picker View wrapper struct BroadcastButtonView: UIViewRepresentable { func makeUIView(context: Context) -> RPSystemBroadcastPickerView { let broadcastPickerView = RPSystemBroadcastPickerView( frame: CGRect(x: 0, y: 0, width: 200, height: 200) ) // Make sure this matches your broadcast extension bundle identifier broadcastPickerView.preferredExtension = "my-extension-bundle-identifier" broadcastPickerView.showsMicrophoneButton = false return broadcastPickerView } func updateUIView(_ uiView: RPSystemBroadcastPickerView, context: Context) { } } The extension SampleHandler: override func broadcastPaused() { print("paused broadcast") // User has requested to pause the broadcast. Samples will stop being delivered. } override func broadcastResumed() { print("resumed broadcast") // User has requested to resume the broadcast. Samples delivery will resume. } override func processSampleBuffer(_ sampleBuffer: CMSampleBuffer, with sampleBufferType: RPSampleBufferType) { print("broadcast received") assetWriter?.processVideoSampleBuffer(sampleBuffer) } Looking forward to any and all help. Information Property list: Information property list for the extension: The capabilities:

After I played the audio for the entity the sound was very low and I wanted to adjust the sound size. No api is found. What should I do if let audio = audioResources { entity.playAudio(audio) }

Hello All, We're going to do a scene now, kind of like a time travel door. When the user selects the scene, the user passes through the door to show the current scene. The changes in the middle need to be more natural. It's even better if you can walk through an immersive space... There is very little information now. How can I start doing this? Is there any information I can refer to thanks

Dear Apple Engineers, I am working on a project in visionOS and need to implement a curved surface effect for video playback, where the width of the surface can be dynamically adjusted. Specifically, I want the video to be displayed on a curved surface (similar to a scroll unfolding), and the user should be able to adjust the width of this surface. I have the following specific questions: How can I implement a curved surface for video playback and ensure the video content is not stretched or distorted on the surface? How can I create a dynamic curved surface (such as a bending plane) in RealityKit or visionOS, where the width can be adjusted by the user? Is it possible to achieve more complex curved surface effects (such as scroll unfolding or bending) using Shaders or other techniques? Thank you very much for your help!

Is this behaviour expected? For example, if I'm using let materials = [SimpleMaterial(color: .red, isMetallic: false)] occlusion works normally, but with let materials = [SimpleMaterial(color: .red.withAlphaComponent(0.5), isMetallic: false)] i can see my cube through real-world objects, like tables, columns, etc. I'm getting the same behaviour if using CustomMaterial from shader and applying customMaterial.blending = .opaque and customMaterial.blending = .transparent(opacity: ) respectively

I want a sentence custom hover effect, not a button. I want a hover effect when you look at one sentence out of many sentences. So I searched for reference videos https://youtu.be/DftRTx1oX6E , https://developer.apple.com/videos/play/wwdc2023/10110/ on apple youtube and visionOS documentation. But I haven't gotten anywhere near my wish feature yet. I respectfully request someone to help me. :)

Platform: iOS18 Tech: RealityView Hi! I was wondering if RealityView now provides ways for their session to persist Anchor data in a world such that the anchor locations in one session can be saved and loaded in a another session that persists the exact same anchor positions. I know that ARWorldMap in ARKit does that, but I was not able to find a way to use it with RealityView. I think it's because RealityView has ARKit under its hood but does not expose the ARKit session info publicly to the client code. So I was wondering if there's a SwiftUI + RealityView approach that can help me to achieve a similar goal: Come back to the same location and see the object in exactly the same place. Thanks!