The new Mac virtual display feature on visionOS 2 offers a curved/panoramic window. I was wondering if this is simply a property that can be applied to a window, or if it involves an immersive mode or SceneKit/RealityKit?

I'm wondering if it's possible to implement object tracking on Vision Pro using the Vision framework of Apple? I see that the Vision documentation offers a variety of classes for computer vision which have a tag "visionOS", but all the example codes in the documentation are only for iOS, iPadOS or macOS. So can those classes also be used for developing Vision Pro apps? If so, how do they get data feed from the camera of Vision Pro?

Currently, visionos is customizing immersive mode in 360-degree full, and I'm looking for a way to adjust it like Apple's basic immersive mode.

Hi, I have a custom object detection CoreML model and I notice something strange when using the model with the Vision framework. I have tried two different approaches as to how to process an image and do inference on the CoreML model. The first one is using the CoreML "raw": initialising the model, getting the input image ready and using the model's .prediction() function to get the models output. The second one is using Vision to wrap the CoreML model in a VNCoreMLModel, creating a VNCoreMLRequest and using the VNImageRequestHandler to actually perform the model inference. The result of the VNCoreMLRequest is of type VNRecognizedObjectObservation. The issue I now face is in the difference in the output of both methods. The first method gives back the raw output of the CoreML model: confidence and coordinates. The confidence is an array with size equal to the number of classes in my model (3 in my case). The second method gives back the boundingBox, confidence and labels. However here the confidence is only the confidence for the most likely class (so size is equal to 1). But the confidence I get from the second approach is quite different from the confidence I get during the first approach. I can use either one of the approaches in my application. However, I really want to find out what is going on and understand how this difference occurred. Thanks!

I am trying to create demo for spatial meeting using persona also refer apple videos, But not getting clear idea about it. Any one could you please guide me step by step process or any code are appreciated for learning.

I am trying to store usdz files with SwiftData for now. I am converting usdz to data, then storing it with SwiftData My model import Foundation import SwiftData import SwiftUI @Model class Item { var name: String @Attribute(.externalStorage) var usdz: Data? = nil var id: String init(name: String, usdz: Data? = nil) { self.id = UUID().uuidString self.name = name self.usdz = usdz } } My function to convert usdz to data. I am currently a local usdz just to test if it is going to work. func usdzData() -> Data? { do { guard let usdzURL = Bundle.main.url(forResource: "tv_retro", withExtension: "usdz") else { fatalError("Unable to find USDZ file in the bundle.") } let usdzData = try Data(contentsOf: usdzURL) return usdzData } catch { print("Error loading USDZ file: \(error)") } return nil } Loading the items @Query private var items: [Item] ... var body: some View { ... ForEach(items) { item in HStack { Model3D(?????) { model in model .resizable() .scaledToFit() } placeholder: { ProgressView() } } } ... } How can I load the Model3D? I have tried: Model3D(data: item.usdz) Gives me the errors: Cannot convert value of type '[Item]' to expected argument type 'Binding<C>' Generic parameter 'C' could not be inferred Both errors are giving in the ForEach. I am able to print the content inside item: ForEach(items) { item in HStack { Text("\(item.name)") Text("\(item.usdz)") } } This above works fine for me. The item.usdz prints something like Optional(10954341 bytes) I would like to know 2 things: Is this the correct way to save usdz files into SwiftData? Or should I use FileManager? If so, how should I do that? Also how can I get the usdz from the storage (SwiftData) to my code and use it into Model3D?

I'm taking my iOS/iPadOS app and converting it so it runs on visionOS. I’m trying to compile my app, build it, for both visionOS and iOS. When I try to build for an iPhone and iPad simulator, I get the following error:  Building for 'iphonesimulator', but realitytool only supports [xros, xrsimulator] I’m thinking I might need to do a # if conditional compilation statement for visionOS so iOS doesn’t try to build lines of code but I can’t for this particular error find out for which file or code I need to do the conditional compilation. Anyone know how to get rid of this error? 

I am doing below code for getting thumbnail from usdz model using the QuickLookThumbnailing, But don't get the proper out. guard let url = Bundle.main.url(forResource: resource, withExtension: withExtension) else{ print("Unable to create url for resource.") return } let request = QLThumbnailGenerator.Request(fileAt: url, size: size, scale: 10.0, representationTypes: .all) let generator = QLThumbnailGenerator.shared generator.generateRepresentations(for: request) { thumbnail, type, error in DispatchQueue.main.async { if thumbnail == nil || error != nil { print(error) }else{ let tempImage = Image(uiImage: thumbnail!.uiImage) print(tempImage) self.thumbnailImage = Image(uiImage: thumbnail!.uiImage) print("=============") } } } } Below Screen Shot for selected model : Below is the thumbnail image, which not come with guitar but get only usdz icon.

I want to get thumbnail image from USDZ model from vision os, But it will get image without material apply. Here is my code import Foundation import SceneKit import SceneKit.ModelIO class ARQLThumbnailGenerator { private let device = MTLCreateSystemDefaultDevice()! /// Create a thumbnail image of the asset with the specified URL at the specified /// animation time. Supports loading of .scn, .usd, .usdz, .obj, and .abc files, /// and other formats supported by ModelIO. /// - Parameters: /// - url: The file URL of the asset. /// - size: The size (in points) at which to render the asset. /// - time: The animation time to which the asset should be advanced before snapshotting. func thumbnail(for url: URL, size: CGSize, time: TimeInterval = 0) -> UIImage? { let renderer = SCNRenderer(device: device, options: [:]) renderer.autoenablesDefaultLighting = true if (url.pathExtension == "scn") { let scene = try? SCNScene(url: url, options: nil) renderer.scene = scene } else { let asset = MDLAsset(url: url) let scene = SCNScene(mdlAsset: asset) renderer.scene = scene } let image = renderer.snapshot(atTime: time, with: size, antialiasingMode: .multisampling4X) self.saveImageFileInDocumentDirectory(imageData: image.pngData()!) return image } func saveImageFileInDocumentDirectory(imageData : Data){ var uniqueID = UUID().uuidString let tempPath = NSSearchPathForDirectoriesInDomains(FileManager.SearchPathDirectory.documentDirectory, FileManager.SearchPathDomainMask.userDomainMask, true) let tempDocumentsDirectory: AnyObject = tempPath[0] as AnyObject let uniqueVideoID = uniqueID + "image.png" let tempDataPath = tempDocumentsDirectory.appendingPathComponent(uniqueVideoID) as String try? imageData.write(to: URL(fileURLWithPath: tempDataPath), options: []) } }

hello I am trying to detect the orientation of text in images. (each image has a label with a number but sometimes the the label is not in the right orientation and I would like two detect these cases and add a prefix to the image files) this code is working well but when the text is upside down it considers that the text is well oriented is it a way to distinguish the difference ? thanks for your help ! import SwiftUI import Vision struct ContentView: View { @State private var totalImages = 0 @State private var processedImages = 0 @State private var rotatedImages = 0 @State private var remainingImages = 0 var body: some View { VStack { Button(action: chooseDirectory) { Text("Choisir le répertoire des images") .padding() } Text("TOTAL: \(totalImages)") Text("TRAITEES: \(processedImages)") Text("ROTATION: \(rotatedImages)") Text("RESTANT: \(remainingImages)") } .padding() } func chooseDirectory() { let openPanel = NSOpenPanel() openPanel.canChooseDirectories = true openPanel.canChooseFiles = false openPanel.allowsMultipleSelection = false openPanel.begin { response in if response == .OK, let url = openPanel.url { processImages(in: url) } } } func processImages(in directory: URL) { DispatchQueue.global(qos: .userInitiated).async { do { let fileManager = FileManager.default let urls = try fileManager.contentsOfDirectory(at: directory, includingPropertiesForKeys: nil) let imageUrls = urls.filter { $0.pathExtension.lowercased() == "jpg" || $0.pathExtension.lowercased() == "png" } DispatchQueue.main.async { self.totalImages = imageUrls.count self.processedImages = 0 self.rotatedImages = 0 self.remainingImages = self.totalImages } for url in imageUrls { self.processImage(at: url) } } catch { print("Error reading contents of directory: \(error.localizedDescription)") } } } func processImage(at url: URL) { guard let image = NSImage(contentsOf: url), let cgImage = image.cgImage(forProposedRect: nil, context: nil, hints: nil) else { return } let request = VNRecognizeTextRequest { (request, error) in if let error = error { print("Error recognizing text: \(error.localizedDescription)") return } if let results = request.results as? [VNRecognizedTextObservation], !results.isEmpty { let orientationCorrect = self.isTextOrientationCorrect(results) if !orientationCorrect { self.renameFile(at: url) DispatchQueue.main.async { self.rotatedImages += 1 } } } DispatchQueue.main.async { self.processedImages += 1 self.remainingImages = self.totalImages - self.processedImages } } let handler = VNImageRequestHandler(cgImage: cgImage, options: [:]) do { try handler.perform([request]) } catch { print("Error performing text recognition request: \(error.localizedDescription)") } } func isTextOrientationCorrect(_ observations: [VNRecognizedTextObservation]) -> Bool { // Placeholder for the logic to check text orientation // This should be implemented based on your specific needs for observation in observations { if let recognizedText = observation.topCandidates(1).first { let boundingBox = observation.boundingBox let angle = atan2(boundingBox.height, boundingBox.width) if abs(angle) > .pi / 4 { return false } } } return true } func renameFile(at url: URL) { let fileManager = FileManager.default let directory = url.deletingLastPathComponent() let newName = "ROTATION_" + url.lastPathComponent let newURL = directory.appendingPathComponent(newName) do { try fileManager.moveItem(at: url, to: newURL) } catch { print("Error renaming file: \(error.localizedDescription)") } } } struct ContentView_Previews: PreviewProvider { static var previews: some View { ContentView() } }

I faced a problem during development that I could not scan Code39 barcode with iPad using Vision. A sample label I used for test has multiple Code39 barcode on it and I could scan almost all barcodes except for specific one. And when I use conventional barcode scanner and free apps to scan barcode, I could scan the barcode with no problem. I failed to scan the barcode only when I use Vision function. Has anyone faced similar situation? Do you know the cause why specific barcode could not be scanned with iPad with Vision?

Hi, I face a problem that I could not scan a specific Code 39 barcode with Vision framework. We have multiple barcode in a label and almost all Code 39 can be scanned, but not for specific one. One more information, regardless the one that is not recognized with Vision can be read by a general barcode scanner. Have anyone faced similar situation? Is there unique condition to make it hard to scan the barcode when using Vision?(size, intensity, etc) Regards,

I'm unable to figure out how to know when my app no longer has focus. ScenePhase will only change when the WindowGroup gets created or closed. UIApplication.didBecomeActive and UIApplication.didEnterBackgroundNotification are not called either when say you move focus to Safari. What's the trick?

xtension Entity { func addPanoramicImage(for media: WRMedia) { let subscription = TextureResource.loadAsync(named:"image_20240425_201630").sink( receiveCompletion: { switch $0 { case .finished: break case .failure(let error): assertionFailure("(error)") } }, receiveValue: { [weak self] texture in guard let self = self else { return } var material = UnlitMaterial() material.color = .init(texture: .init(texture)) self.components.set(ModelComponent( mesh: .generateSphere(radius: 1E3), materials: [material] )) self.scale *= .init(x: -1, y: 1, z: 1) self.transform.translation += SIMD3(0.0, -1, 0.0) } ) components.set(Entity.WRSubscribeComponent(subscription: subscription)) } func updateRotation(for media: WRMedia) { let angle = Angle.degrees( 0.0) let rotation = simd_quatf(angle: Float(angle.radians), axis: SIMD3<Float>(0, 0.0, 0)) self.transform.rotation = rotation } struct WRSubscribeComponent: Component { var subscription: AnyCancellable } } case .failure(let error): assertionFailure("(error)") Thread 1: Fatal error: Error Domain=MTKTextureLoaderErrorDomain Code=0 "Image decoding failed" UserInfo={NSLocalizedDescription=Image decoding failed, MTKTextureLoaderErrorKey=Image decoding failed}

Does anyone have a ready-made script/shortcut like the one shown in the video?

For example: we use DocKit for birdwatching, so we have an unknown field distance and direction. Distance = ? Direction = ? For example, the rock from which the observation is made. The task is to recognize the number of birds caught in the frame, add a detection frame and collect statistics. Question: What is the maximum number of frames processed with custom object recognition? If not enough, can I do the calculations myself and transfer to DokKit for fast movement?

I want to set collection in curve view with fix paging in vision pro, How can i do?

It seems like eye tracking comes into conflict with pointing devices and the cursor will jump to where the user looks at, which makes trackpad control very janky. Is there a way to filter eye tracking and just enable pointing devices?

I want to get only spatial video while open the Photo library in my app. How can I achieve? One more thing, If I am selecting any video using photo library then how to identify selected video is Spatial Video or not? self.presentPicker(filter: .videos) /// - Tag: PresentPicker private func presentPicker(filter: PHPickerFilter?) { var configuration = PHPickerConfiguration(photoLibrary: .shared()) // Set the filter type according to the user’s selection. configuration.filter = filter // Set the mode to avoid transcoding, if possible, if your app supports arbitrary image/video encodings. configuration.preferredAssetRepresentationMode = .current // Set the selection behavior to respect the user’s selection order. configuration.selection = .ordered // Set the selection limit to enable multiselection. configuration.selectionLimit = 1 let picker = PHPickerViewController(configuration: configuration) picker.delegate = self present(picker, animated: true) } `func picker(_ picker: PHPickerViewController, didFinishPicking results: [PHPickerResult]) { picker.dismiss(animated: true) { // do something on dismiss } guard let provider = results.first?.itemProvider else {return} provider.loadFileRepresentation(forTypeIdentifier: "public.movie") { url, error in guard error == nil else{ print(error) return } // receiving the video-local-URL / filepath guard let url = url else {return} // create a new filename let fileName = "\(Int(Date().timeIntervalSince1970)).\(url.pathExtension)" // create new URL let newUrl = URL(fileURLWithPath: NSTemporaryDirectory() + fileName) print(newUrl) print("===========") // copy item to APP Storage //try? FileManager.default.copyItem(at: url, to: newUrl) // self.parent.videoURL = newUrl.absoluteString } }`

App A want to know if iPhone user is looking at the photo one of the family member and display some information related to that user?