I am using ArKit to create an augmented reality application in Unity. Following the addition of an object reference object Because it tracks the object in front of it slowly and inaccurately, the application does not update the screen quickly. How can I track objects more quickly?

I'm working on training an object tracking model in CreateML for visionOS that has fan blades on it and looking to try to train while ignoring a section of the geometry. When I train currently, it can detect the object if the fan blades are in the same orientation as when scanned but if they move it struggles. I see there is an "objects to avoid" data source that can be added but upon reading the description, I don't think it does what I'm needing but I could be wrong. Is there anyway to have the training ignore a part of the geometry that has a significant effect on the silhouette of the object?

I'm seeking insight on why the new VisionOS Barcode Scanning API is categorized as an Enterprise API and restricted only for proprietary and in-house apps. I understand Apple's focus on privacy and I can see how this restriction could make sense for other Enterprise APIs like main camera access and passthrough screen capture. Why is barcode scanning restricted from open apps? What makes barcode scanning more of a risk to privacy versus the unrestricted APIs for object tracking, image tracking, or hand tracking?

Hi all, I tried the "isSpatialVideoCaptureEnabled" with AVCaptureMovieFileOutput mentioned in WWDC24: Build compelling spatial photo and video experiences, and it works. But there are some issues and questions: Below codes, the change.newValue always nil so the code seems not work. let observation = videoDevice.observe(\.spatialCaptureDiscomfortReasons) { (device, change) in guard let newValue = change.newValue else { return } if newValue.contains(.subjectTooClose) { // Guide user to move back } if newValue.contains(.notEnoughLight) { // Guide user to find a brighter environment } } AVCaptureMovieFileOutput is support spatial video capturing. May I ask if AVCaptureVideoDataOutput will also support spatial video capturing?

FYI. The source code of the FindSurface demo app for Apple Vision Pro (visionOS) is available now. The Swift package of FindSurface™ library is required to build the source code into the demo app. https://github.com/CurvSurf/FindSurface-visionOS After starting the app, the floating panels (below) will appear on your right side, and you will see wireframe meshes that approximately describe your environments. Performing a spatial tap (pinching with your thumb and index finger) with staring at a location on the meshes will invoke FindSurface, with an indicator (blue disk) appearing on the surface you've gazed. Voice commands: “Tap” – Spatial tap (gazing & pinching). Invoke FindSurface. “Tap plane” – Plane selection. “Tap sphere” or “Tap ball” – Sphere selection. “Tap cylinder” – Cylinder selection. “Tap cone” – Cone selection. “Tap torus” or “Tap donut” – Torus selection. “Tap accuracy” or “Tap measurement accuracy” – Accuracy selection. “Tap mean distance”, “Tap average distance”, or “Tap distance” – Avg. Distance selection. “Tap touch radius” or “Tap seed radius” – Touch Radius selection. “Tap Inlier” – “Show inlier points” toggle. “Tap outline” – “Show geometry outline” toggle. “Tap clear” – “Clear Scene” click.

Hello everyone, It seems that Vision Pro supports 6DoF tracking, but is it possible to switch to 3DoF tracking? The reason for my question is that I would like to use it while riding in a car, but it seems that the 6DoF tracking is not working well in this situation. I was wondering if switching to 3DoF tracking might solve the issue. Note: I am using the travel mode.

I'm developing an app where a user can bring a video or content from a WKWebView into an immersive space using SwiftUI attachments on a RealityView. This works just fine, but I'm having some trouble configuring how the audio from the web content should sound in an immersive space. When in windowed mode, content playing sounds just fine and very natural. The spatial audio effect with head tracking is pronounced and adds depth to content with multichannel or Dolby Atmos audio. When I move the same web view into an immersive space however, the audio becomes excessively echoey, as if a large amount of reverb has been put onto the audio. The spatial audio effect is also decreased, and while still there, is no where near as immersive. I've tried the following: Setting all entities in my space to use channel audio, including the web view attachment. for entity in content.entities { entity.channelAudio = ChannelAudioComponent() entity.ambientAudio = nil entity.spatialAudio = nil } Changing the AVAudioSessionSpatialExperience: And I've also tried every soundstage size and anchoring strategy, large works the best, but doesn't remove that reverb. let experience = AVAudioSessionSpatialExperience.headTracked( soundStageSize: .large, anchoringStrategy: .automatic ) try? AVAudioSession.sharedInstance().setIntendedSpatialExperience(experience) I'm also aware of ReverbComponent in visionOS 2 (which I haven't updated to just yet), but ideally I need a way to configure this for visionOS 1 users too. Am I missing something? Surely there's a way for developers to stop the system messing with the audio and applying these effects? A few of my users have complained that the audio sounds considerably worse in my cinema immersive space compared to in a window.

Hi all, Currently working on a shareplay feature where users pull data from a remote source and are able to share it in a volumetric window with others in the facetime call. However, I am running into an issue where the group activity/session seems to be throwing an error on the recipient of the journal's attachment with the description of notSupported. As I understand it, we use GroupSessionJournal for larger pieces of data like images (like in the Drawing Together example) and in my case 3d models. The current flow goes as follows: User will launch the app and fetch a model from remote. User can start a shareplay instance in which the system captures the volumetric window for users to join and see. At this point, only the original user can see the model. The user can press a button to share this model with the other participants using /// modelData is serialized `Data` try await journal.add(modelData) In the group session configuration, I already have a task listening for for await attachments in journal.attachments { for attachment in attachments { ... } } This task attempts to load data via the following code: let modelData = try await attachment.load(Data.self) /// this is where the error is thrown: `notSupported` I expect the attachment.load(Data.self) call to properly deliver the model data, but instead I am receiving this error. I have also attempted to wrap the model data within an enclosing struct that has a name and data property and conform the enclosing struct to Transferable but that continued to throw the notSupported error. Is there something I'm doing wrong or is this simply a bug in the GroupSessionJournal? Please let me know if more information is required for debugging and resolution. Thanks!

At the end of the WWDC 2023 sessoion https://developer.apple.com/wwdc23/10111 the session talks about implementing Virtual Hands. However the Code shown in the Video is not correct anymore with the latest version of VisionOS and also the example “SpaceGloves“ Entity referenced in the Video is not documented and there is no example project resource to reference either. I have been trying for the last two weeks to implement the same basic example shown in this video, including skinning and rigging my own hand meshes But have had significant issues doing so and found no working code/usdz examples to reference across different internet resource. Is it possible for a working project/code example including USDZ files for the Hand Meshes to be provided in order to test a working example of this as a starting point for building my own virtual hands? Thanks for your help.

Is there a way to access the coordinates of where the camera is while scanning the room with Roomplan?

Hi, I read online that to downgrade from Vision OS 2 developer beta back to Vision OS 1 - we need the developer strap. My request for the strap is still under review, is there a way to downgrade without the strap if the need arises?

I've got an Immersive scene that I want to be able to bring additional users into via SharePlay where each user would be able to see (and hopefully interact) with the Immersive scene. How does one implement that?

In Progressive mode, you can turn the digital crown which will reveal your environment by limiting/expanding the field of view of your Immersive scene. I'm trying to create a different sort of behavior where your Immersive scene remains in 360 mode but adjusting a dial (doesn't have to be the crown, it could be an in-app dial/slider) adjusts the transparency of the scene. My users aren't quite satisfied with the native features that help ensure you aren't about to run into a wall or furniture and want a way of quickly adjusting the transparency on the fly. Is that possible?

Hi, I'm very new to 3D and am currently porting a SwiftUI iOS app to visionOS 2.0. I saw WWDC24 feature Blender in multiple spatial videos, and have begun integrating Blender models and animations into my VisionOS app (I would also like to integrate skeletons and programmatic rigging, more on that later). I'm wondering if there are “Best Practices” for this workflow - from Blender to USD to RCP 2.0 to visionOS 2 in Xcode. I’ve cobbled together the following that has some obvious holes: I’ve been able to find some pre-rigged and pre-animated models online that can serve as a great starting point. As a reference, here is a free model from SketchFab - a simple rigged skeleton with 6 built in animations: https://sketchfab.com/3d-models/skeleton-character-low-poly-8856e0138f424d68a8e0b40e185951f6 When exporting to USD from Blender, I haven’t been able to export more than one animation per USD file. Is there a workflow to export multiple animations in a single USDC file, or is this just not possible? As a temporary workaround, here is a python script I’ve been using to loop through all Blender animations, and export a model for each animation: import bpy import os # Set the directory where you want to save the USD files output_directory = “/path/to/export” # Ensure the directory exists if not os.path.exists(output_directory): os.makedirs(output_directory) # Function to export current scene as USD def export_scene_as_usd(output_path, start_frame, end_frame): bpy.context.scene.frame_start = start_frame bpy.context.scene.frame_end = end_frame # Export the scene as a USD file bpy.ops.wm.usd_export( filepath=output_path, export_animation=True ) # Save the current scene name original_scene = bpy.context.scene.name # Iterate through each action and export it as a USD file for action in bpy.data.actions: # Create a new scene for each action bpy.context.window.scene = bpy.data.scenes[original_scene].copy() new_scene = bpy.context.scene # Link the action to all relevant objects for obj in new_scene.objects: if obj.animation_data is not None: obj.animation_data.action = action # Determine the frame range for the action start_frame, end_frame = action.frame_range # Export the scene as a USD file output_path = os.path.join(output_directory, f"{action.name}.usdc") export_scene_as_usd(output_path, int(start_frame), int(end_frame)) # Delete the temporary scene to free memory bpy.data.scenes.remove(new_scene) print("Export completed.") I have also been able to successfully export rigging armatures as a single Skeleton - each “bone” showing getting imported into Reality Composer Pro 2.0 when exporting/importing manually. I would like to have all of these animations available in a single scene to be used in a RealityView in visionOS - so I have placed all animation models in a RCP scene and created named Timeline Action animations for each, showing the correct model and hiding the rest when triggering specific animations. I apply materials/textures to each so they appear the same, using Shader Graph. Then in SwiftUI I use notifications (as shown here - https://forums.developer.apple.com/forums/thread/756978) to trigger each RCP Timeline Action animation from code. Two questions: Is there a better way than to have multiple models of the same skeleton - each with a different animation - in a scene to be able to trigger multiple animations? Or would this require recreating Blender animations using skeleton rigging and keyframes from within RCP Timelines? If I want to programmatically create custom animations and move parts of the skeleton/armatures - do I need to do this by defining custom components in RCP, using IKRig and define movement of each of the “bones” in Xcode? I’m looking for any tips/tricks/workflow from experienced engineers or 3D artists that can create a more efficient/optimized workflow using Blender, USD, RCP 2 and visionOS 2 with SwiftUI. Thanks so much, I appreciate any help! I am very excited about all the new tools that keep evolving to make spatial apps really fun to build!

Hello! I have developed an application using Unity that runs on the visionPro. I have correctly built it, installed it on the device and it works. The spatial computing application continue to work after several days (pretty normal, I launch the app and it works. It doesn't use any external services). After several weeks, like a month or so, I launch again the same app but it's not working anymore. The only way I have to make it work again is to rebuild and reinstall it again. What am I missing here? Why an application built and installed few weeks ago suddenly stops working on the VisionPro?

Hello, I am trying to make an app that involves room scanning and then placing of imaginary objects in the room. I had two questions about the specifics behind this. Is it possible for Roomplan to include the ceiling when scanning the room? Is it possible to place objects in AR while Room plan is running, or is it necessary to wait until after the scan is done?

Hello, I want to capture video from Vision Pro in the Vision OS app. I am referring to the (https://developer.apple.com/videos/play/wwdc2024/10139/) Apple video and their code. step like below import ARKit com.apple.developer.arkit.main-camera-access.allow = true in info.plist Do below code func loadCameraFeed() async { // Main Camera Feed Access Example let formats = CameraVideoFormat.supportedVideoFormats(for: .main, cameraPositions:[.left]) let cameraFrameProvider = CameraFrameProvider() var arKitSession = ARKitSession() var pixelBuffer: CVPixelBuffer? await arKitSession.queryAuthorization(for: [.cameraAccess]) do { try await arKitSession.run([cameraFrameProvider]) } catch { return } guard let cameraFrameUpdates = cameraFrameProvider.cameraFrameUpdates(for: formats[0]) else { return } print(cameraFrameUpdates) for await cameraFrame in cameraFrameUpdates { print(cameraFrame) guard let mainCameraSample = cameraFrame.sample(for: .left) else { continue } pixelBuffer = mainCameraSample.pixelBuffer } } I want to convert "pixelBuffer" into video streaming and show it in a frame like iOS. Please guide me on how to achieve my next step. I am blank after this code.

For context, we have a fully immersive application running on visions. The application starts with a standard View/menu, and when the user clicks an option, it takes you to fully immersive mode with a small floating toolbar window that you can move and interact with as you move around the virtual space. When the user clicks the small x button below the window, we intercept the scenePhase .background event and handle exiting immersive mode and displaying the main menu. This all works fine. The problem happens if the user turns around and doesn't look at the floating window for a couple of minutes. The system decides that the window should go into the background, and the same scenePhase background event is called - causing the system to exit immersive mode without warning. There seems to be no way of preventing this, and no distinction between this and the user clicking the close button. Is there a reliable way to detect if the user intentionally clicks the close button vs the window from going into the background through lack of use? onDisappear doesn't trigger. thanks in advance

I'm having issue with loading 1.2GB USDZ file with visionOS Here's the details the file is download via backend api file is download to document directory FileManager.default.url(for: .documentDirectory, in: .userDomainMask, appropriateFor: nil, create: true) when loading the asset, it took almost 30s to load the asset Loaded usd((extension in RealityFoundation):RealityKit.Entity.LoadStatistics.USDLoader.rio) in 29.24642503261566 seconds loading asset code let model = try await Entity(contentsOf: assetUrl) USDZ file is exported from RealityComposerPro Did I make any mistake on the flow or is there any other approach to decrease the loading time?

Hello I've just updated my Vision Pro to the newest and greatest 2.0, and I see that the way to call out control center has been changed to the hand gesture which I'm assuming that is powered by computer vision? using the cams, for me there is a use case of watching apple TV shows at night where my gf would like to have the lights turned off, and that is the time where it fails, the new method is greate and the responsiveness are crazily good, but I would like this to be a toggle so that we can select our own method? like for example during the day where lights are sufficient we use the new gestrue recogn way, and when low light we can switch it back to look up? or as a fellow programmer who are just learning, I think it is possible to just make it automatic toggle, whenever the lighting condition can't make the hand gestrue work, hope to see that fixed :) cheers