I use DataScannerViewController to scan barcode and text recognize, and then get the AVCaptureDevice to use torch on/off, but DataScannerViewController will stop scanning. DataScannerViewController has no related API to get AVCaptureDevice to use torch. Expected: Could use AVCaptureDevice to turn on/off torch, at the same time DataScannerViewController could scan.

In visionOS, how can I use vision in machine learning to recognize hand gestures? After all, currently visionOS does not provide any image frame data.

I have a live text implementation on the following LiveTextImageView. However, after the view loads and the analyze code is run, none of the delegate methods fire when I interact with the Live View. Selecting text does not fire the textSelectionDidChange method, nor does highlightSelectedItemsDidChange fire when the live text button in the bottom right is pressed. I tried a few different implementations, including an approach where the delegate was defined on a separate class. I am running this on a iPhone 12 Pro I recently updated to 17.0.3. My goal is to be able to provide additional options to the user beyond the default live-text overlay options, after identifying when they have finished selecting text. // // LiveTextImageView.swift // import UIKit import SwiftUI import VisionKit class ImageAnalyzerWrapper { static let shared = ImageAnalyzer() private init() { } } struct LiveTextImageViewRepresentable: UIViewRepresentable { var image: UIImage func makeUIView(context: Context) -> LiveTextImageView { return LiveTextImageView(image: image) } func updateUIView(_ uiView: LiveTextImageView, context: Context) { } } class LiveTextImageView: UIImageView, ImageAnalysisInteractionDelegate, UIGestureRecognizerDelegate { var capturedSelectedText: String? let analyzer = ImageAnalyzerWrapper.shared let interaction = ImageAnalysisInteraction() init(image: UIImage) { super.init(frame: .zero) self.image = image let photoWrapper = PhotoWrapper(rawPhoto: image) let resizedPhoto = photoWrapper.viewportWidthCroppedPhoto(padding: 40) self.image = resizedPhoto self.contentMode = .scaleAspectFit self.addInteraction(interaction) interaction.preferredInteractionTypes = [] interaction.analysis = nil analyzeImage() } required init?(coder: NSCoder) { fatalError("init(coder:) has not been implemented") } func analyzeImage() { if let image = self.image { Task { let configuration = ImageAnalyzer.Configuration([.text]) do { let analysis = try await analyzer.analyze(image, configuration: configuration) self.addInteraction(interaction) interaction.delegate = self interaction.analysis = analysis interaction.preferredInteractionTypes = .textSelection } catch { print("Error in live image handling") } } } } func interaction( _ interaction: ImageAnalysisInteraction, highlightSelectedItemsDidChange highlightSelectedItems: Bool) async { print("Highlighted items changed") } func interaction(_ interaction: ImageAnalysisInteraction, shouldBeginAt point: CGPoint, for interactionType: ImageAnalysisInteraction.InteractionTypes) async -> Bool { return interaction.hasInteractiveItem(at: point) || interaction.hasActiveTextSelection } func textSelectionDidChange(_ interaction: ImageAnalysisInteraction) async { print("Changed!") if #available(iOS 17.0, *) { capturedSelectedText = interaction.text print(capturedSelectedText ?? "") } } }

Eroor

Hi Everyone, I'm having a strange crash on App launch with iOS16 when I have a reference to an iOS17 only framework in my code. Even if I wrap the code in #available, I still get the crash on launch; and the code isn't even called yet... just the existence of it causes the crash. Pretty strange I thought? The framework is VisionKit, and the code that causes the crash is if #available(iOS 17, *) { // .imageSubject is iOS17 only - but this causes // a crash on launch in iOS16 even with the #available check interaction.preferredInteractionTypes = .imageSubject } The crash is: Referenced from: <91ED5216-D66C-3649-91DA-B31C0B55DDA1> /private/var/containers/Bundle/Application/78FD9C93-5657-4FF5-85E7-A44B60717870/XXXXXX.app/XXXXXX Expected in: <AF01C435-3C37-3C7C-84D9-9B5EA3A59F5C> /System/Library/Frameworks/VisionKit.framework/VisionKit Any thoughts anyone?? I know the .imageSubject is iOS17 only, but the #available should catch it - no? Any why does it crash immediatley on launch, when that code is not even called? Odd!

Hi team, We have an iOS app. Since July 15, 2022, our users met a kind of app crash due to an invalid memory fetch. The time is when Apple released iOS 16 beta officially. After Sep 12, crash count started to increase drastically. The time is Apple released iOS 16 officially. Crash backtrace can be seen as follows. Thread 14 Crashed: 0 libsystem_platform.dylib 0x00000001f8810930 _platform_memmove + 96 1 CoreGraphics 0x00000001adb64104 CGDataProviderCreateWithCopyOfData + 20 2 CoreGraphics 0x00000001adb4cdb4 CGBitmapContextCreateImage + 172 3 VisionKitCore 0x00000001ed813f10 -[VKCRemoveBackgroundResult _createCGImageFromBGRAPixelBuffer:cropRect:] + 348 4 VisionKitCore 0x00000001ed813cc0 -[VKCRemoveBackgroundResult createCGImage] + 156 5 VisionKitCore 0x00000001ed8ab6f8 __vk_cgImageRemoveBackgroundWithDownsizing_block_invoke + 64 6 VisionKitCore 0x00000001ed881474 __63-[VKCRemoveBackgroundRequestHandler performRequest:completion:]_block_invoke.5 + 436 7 MediaAnalysisServices 0x00000001eec58968 __92-[MADService performRequests:onPixelBuffer:withOrientation:andIdentifier:completionHandler:]_block_invoke.38 + 400 8 CoreFoundation 0x00000001abff0a14 __invoking___ + 148 9 CoreFoundation 0x00000001abf9cf2c -[NSInvocation invoke] + 428 10 Foundation 0x00000001a6464d38 __NSXPCCONNECTION_IS_CALLING_OUT_TO_REPLY_BLOCK__ + 16 11 Foundation 0x00000001a64362fc -[NSXPCConnection _decodeAndInvokeReplyBlockWithEvent:sequence:replyInfo:] + 520 12 Foundation 0x00000001a6a10f44 __88-[NSXPCConnection _sendInvocation:orArguments:count:methodSignature:selector:withProxy:]_block_invoke_5 + 188 13 libxpc.dylib 0x00000001f89053e4 _xpc_connection_reply_callout + 124 14 libxpc.dylib 0x00000001f88f8580 _xpc_connection_call_reply_async + 88 15 libdispatch.dylib 0x00000001b340205c _dispatch_client_callout3 + 20 16 libdispatch.dylib 0x00000001b341ff58 _dispatch_mach_msg_async_reply_invoke + 344 17 libdispatch.dylib 0x00000001b340956c _dispatch_lane_serial_drain + 376 18 libdispatch.dylib 0x00000001b340a214 _dispatch_lane_invoke + 436 19 libdispatch.dylib 0x00000001b3414e10 _dispatch_workloop_worker_thread + 652 20 libsystem_pthread.dylib 0x00000001f88a4df8 _pthread_wqthread + 288 21 libsystem_pthread.dylib 0x00000001f88a4b98 start_wqthread + 8 Last but not the least. The users who met this kind of app crash use iOS16+. We think this crash is related to iOS 16 SDK. We're appreciate that you can provide some clues how to fix this kind of crash.

I notice that some WWDC sessions have a code tab (in addition to Overview and Transcript) but this session 10176 does not. I tried what code I could see on the video but it's obviously not a complete project. It would help if the authors of the Session 10176 video could add the code to the session.

var accessibilityComponent = AccessibilityComponent() accessibilityComponent.isAccessibilityElement = true accessibilityComponent.traits = [.button, .playsSound] accessibilityComponent.label = "Cloud" accessibilityComponent.value = "Grumpy" cloud.components[AccessibilityComponent.self] = accessibilityComponent // ... var isHappy: Bool { didSet { cloudEntities[id].accessibilityValue = isHappy ? "Happy" : "Grumpy" } }

I just grabbed the portal code made available for testing and ran into this error when trying to run in simulator Vision Pro Thread 1: Fatal error: SwiftUI Scene ImmersiveSpace requires a UISceneSessionRole of "UISceneSessionRoleImmersiveSpaceApplication" for key UIApplicationPreferredDefaultSceneSessionRole in the Application Scene Manifest.

Using VNDocumentCameraViewController, if the document is automatically scanned, it cannot be obtained in the func documentCameraViewController (_ controller: VNDocumentCameraViewController, didFinishWith scan: VNDocumentCameraScan) {}. If manual photography is used, data can be obtained May I ask how to solve it?

I trying to scan credit card but i am getting some issue with printed( Pressed ) number on card and some dark background with dark card numbers as well not scanning. Please help me to smooth scan for every cards. Thanks.

Hi, do you know if it's possible to handle the flashlight inside DataScannerViewController? I tried with AVCaptureDevice.DiscoverySession, but clearly when torchmode is on DataScannerViewController freeze... Thanks Luca

Hi, When using VNFeaturePrintObservation and then computing the distance using two images, the values that it returns varies heavily. When two identical images (same image file) is inputted into function (below) that I have used to compare the images, the distance does not return 0 while it is expected to, since they are identical images. Also, what is the upper limit of computeDistance? I am trying to find the percentage similarity between the two images. (Of course, this cannot be done unless the issue above is resolved). Code that I have used is below func featureprintObservationForImage(image: UIImage) -> VNFeaturePrintObservation? {     let requestHandler = VNImageRequestHandler(cgImage: image.cgImage!, options: [:])     let request = VNGenerateImageFeaturePrintRequest()     request.usesCPUOnly = true // Simulator Testing     do {       try requestHandler.perform([request])       return request.results?.first as? VNFeaturePrintObservation     } catch {       print("Vision Error: \(error)")       return nil     }   }   func compare(origImg: UIImage, drawnImg: UIImage) -> Float? {     let oImgObservation = featureprintObservationForImage(image: origImg)     let dImgObservation = featureprintObservationForImage(image: drawnImg)     if let oImgObservation = oImgObservation {       if let dImgObservation = dImgObservation {         var distance: Float = -1         do {           try oImgObservation.computeDistance(&distance, to: dImgObservation)         } catch {           fatalError("Failed to Compute Distance")         }         if distance == -1 {           return nil         } else {           return distance         }       } else {         print("Drawn Image Observation found Nil")       }     } else {       print("Original Image Observation found Nil")     }     return nil   } Thanks for all the help!

Hello, I'm doing a iOS app and I'm trying to find a way to extract programmatically a person from his identity picture (and to leave behind the background) I'm watching WWDC "Lift subjects from images in your app" video (a really cool feature) and i'm wondering if this feature would be possible programmatically, without the need of a human person interaction. Thank you.

In VisionOS is it possible to detect when a user is touching a physical surface in the real world and also to project 2D graphics on that surface? So imagine a windowless 2D app that is projected onto a surface, essentially turning a physical wall, table, etc. into a giant touchscreen? So kinda like this: https://appleinsider.com/articles/23/06/23/vision-pro-will-turn-any-surface-into-a-display-with-touch-control But I want every surface in the room to be touchable and be able to display 2D graphics on the face of that surface and not floating in space. So essentially turning every physical surface in the room into a UIView. Thanks!

I'm using VNDocumentViewController to scan some documents which is working fine. But not infrequently, the app becomes slow after dismissing the VNDocumentViewController. There is no VNDocumentViewController instance allocated, but according to the allocations gathered by instruments, there is a ICDocCamViewController still living and using between 200 and 300 MB. I guess that ICDocCamViewController is an internal component of VNDocumentViewController. Are there any known issues? Unfortunately, I do not see any way to free ICDocCamViewController. 281.37 MB 93.1% 104708 start_wqthread 281.37 MB 93.1% 104708 _pthread_wqthread 256.04 MB 84.7% 42747 _dispatch_workloop_worker_thread 256.03 MB 84.7% 42564 _dispatch_lane_invoke 256.03 MB 84.7% 42564 _dispatch_lane_serial_drain 249.91 MB 82.7% 22258 _dispatch_client_callout 237.01 MB 78.4% 8562 _dispatch_call_block_and_release 236.50 MB 78.3% 3836 __77-[ICDocCamViewController saveCapturedImage:metaData:rects:completionHandler:]_block_invoke_3 236.50 MB 78.3% 3836 -[ICDocCamViewController cropAndFilterImage:rects:filterType:] 236.49 MB 78.3% 3678 +[ICDocCamImageFilters filteredImage:orientation:imageFilterType:] 236.49 MB 78.3% 3678 +[ICDocCamImageFilters colorDocument:orientation:] 236.47 MB 78.3% 3476 -[CIContext(createCGImage) createCGImage:fromRect:] 236.47 MB 78.3% 3476 -[CIContext(_createCGImageInternal) _createCGImage:fromRect:format:premultiplied:colorSpace:deferred:renderCallback:] 236.40 MB 78.2% 2096 -[CIContext(CIRenderDestination) startTaskToRender:fromRect:toDestination:atPoint:error:] 236.40 MB 78.2% 2096 -[CIContext(CIRenderDestination) _startTaskToRender:toDestination:forPrepareRender:forClear:error:] 236.40 MB 78.2% 2096 CI::RenderToBitmap::render(CI::Image*, CI::Context*) const 236.40 MB 78.2% 2096 CI::image_get_bitmap(CI::Context*, CI::Image*, CGRect, CGColorSpace*, CI::Bitmap*, CI::RenderDestination const*) 236.33 MB 78.2% 716 CI::tile_node_graph(CI::Context*, CI::RenderDestination const*, char const*, CI::Node*, CGRect const&, CI::PixelFormat, CI::swizzle_info const&, CI::TileTask* (CI::ProgramNode*, CGRect) block_pointer) 236.33 MB 78.2% 716 CI::recursive_tile(CI::RenderTask*, CI::Context*, CI::RenderDestination const*, char const*, CI::Node*, CGRect const&, CI::PixelFormat, CI::swizzle_info const&, CI::TileTask* (CI::ProgramNode*, CGRect) block_pointer) 236.26 MB 78.2% 212 invocation function for block in CI::image_get_bitmap(CI::Context*, CI::Image*, CGRect, CGColorSpace*, CI::Bitmap*, CI::RenderDestination const*) 236.26 MB 78.2% 212 CI::Context::render(CI::ProgramNode*, CGRect const&) 236.26 MB 78.2% 212 CI::Context::recursive_render(CI::TileTask*, CI::roiKey const&, CI::Node*, bool) 236.26 MB 78.2% 211 CI::Context::recursive_render(CI::TileTask*, CI::roiKey const&, CI::Node*, bool) 236.24 MB 78.2% 196 CI::Context::recursive_render(CI::TileTask*, CI::roiKey const&, CI::Node*, bool) 236.23 MB 78.2% 181 CI::Context::recursive_render(CI::TileTask*, CI::roiKey const&, CI::Node*, bool) 236.21 MB 78.2% 166 CI::Context::recursive_render(CI::TileTask*, CI::roiKey const&, CI::Node*, bool) 236.19 MB 78.2% 151 CI::Context::recursive_render(CI::TileTask*, CI::roiKey const&, CI::Node*, bool) 17.02 KB 0.0% 14 CI::MetalTextureManager::create_intermediate(CI::IntermediateDescriptor const&, unsigned long long, CGRect const&, unsigned long, unsigned long, bool) 128 Bytes 0.0% 1 CI::MetalContext::render_intermediate_node(CI::TileTask*, CI::ProgramNode*, CGRect const&, CI::intermediate_t*, bool, void () block_pointer) 17.02 KB 0.0% 14 CI::MetalTextureManager::create_intermediate(CI::IntermediateDescriptor const&, unsigned long long, CGRect const&, unsigned long, unsigned long, bool) 128 Bytes 0.0% 1 CI::MetalContext::render_intermediate_node(CI::TileTask*, CI::ProgramNode*, CGRect const&, CI::intermediate_t*, bool, void () block_pointer) 17.02 KB 0.0% 14 CI::MetalTextureManager::create_intermediate(CI::IntermediateDescriptor const&, unsigned long long, CGRect const&, unsigned long, unsigned long, bool) 128 Bytes 0.0% 1 CI::MetalContext::render_intermediate_node(CI::TileTask*, CI::ProgramNode*, CGRect const&, CI::intermediate_t*, bool, void () block_pointer) 17.02 KB 0.0% 14 CI::MetalTextureManager::create_intermediate(CI::IntermediateDescriptor const&, unsigned long long, CGRect const&, unsigned long, unsigned long, bool) 128 Bytes 0.0% 1 CI::MetalContext::render_intermediate_node(CI::TileTask*, CI::ProgramNode*, CGRect const&, CI::intermediate_t*, bool, void () block_pointer) 128 Bytes 0.0% 1 CI::MetalContext::render_root_node(CI::TileTask*, CI::ProgramNode*, CGRect const&, void () block_pointer, void () block_pointer) 69.00 KB 0.0% 504 CI::gather_rois_for_program_graph(CI::Context*, char const*, CI::ProgramNode*, CGRect) 71.72 KB 0.0% 1380 CI::prepare_initial_graph(CI::Context*, char const*, CI::Image*, CI::RenderDestination const*, CGRect, CGColorSpace*, CI::PixelFormat, CI::swizzle_info, CI::Affine const&, bool, CI::TextureDescriptor*) 71.72 KB 0.0% 1380 CI::prepare_initial_graph(CI::Context*, char const*, CI::Image*, CI::RenderDestination const*, CGRect, CGColorSpace*, CI::PixelFormat, CI::swizzle_info, CI::Affine const&, bool, CI::TextureDescriptor*) 18.34 KB 0.0% 202 -[CIPaperWash outputImage]

I work on an iOS app that displays images that often contain text, and I'm adding support for ImageAnalysisInteraction as described in this WWDC 2022 session. I have gotten as far as making the interaction show up and being able to select text and get the system selection menu, and even add my own action to the menu via the buildMenuWithBuilder API. But what I really want to do with my custom action is get the selected text and do a custom lookup-like thing to check the text against other content in my app. So how do I get the selected text from an ImageAnalysisInteraction on a UIImageView? The docs show methods to check if there is selected text, but I want to know what the text is.

With the release of Xcode 13, a large section of my vision framework processing code became errors and cannot compile. All of these have became deprecated. This is my original code:  do {       // Perform VNDetectHumanHandPoseRequest       try handler.perform([handPoseRequest])       // Continue only when a hand was detected in the frame.       // Since we set the maximumHandCount property of the request to 1, there will be at most one observation.       guard let observation = handPoseRequest.results?.first else {         self.state = "no hand"         return       }       // Get points for thumb and index finger.       let thumbPoints = try observation.recognizedPoints(forGroupKey: .handLandmarkRegionKeyThumb)       let indexFingerPoints = try observation.recognizedPoints(forGroupKey: .handLandmarkRegionKeyIndexFinger)       let middleFingerPoints = try observation.recognizedPoints(forGroupKey: .handLandmarkRegionKeyMiddleFinger)       let ringFingerPoints = try observation.recognizedPoints(forGroupKey: .handLandmarkRegionKeyRingFinger)       let littleFingerPoints = try observation.recognizedPoints(forGroupKey: .handLandmarkRegionKeyLittleFinger)       let wristPoints = try observation.recognizedPoints(forGroupKey: .all)               // Look for tip points.       guard let thumbTipPoint = thumbPoints[.handLandmarkKeyThumbTIP],          let thumbIpPoint = thumbPoints[.handLandmarkKeyThumbIP],          let thumbMpPoint = thumbPoints[.handLandmarkKeyThumbMP],          let thumbCMCPoint = thumbPoints[.handLandmarkKeyThumbCMC] else {         self.state = "no tip"         return       }               guard let indexTipPoint = indexFingerPoints[.handLandmarkKeyIndexTIP],          let indexDipPoint = indexFingerPoints[.handLandmarkKeyIndexDIP],          let indexPipPoint = indexFingerPoints[.handLandmarkKeyIndexPIP],          let indexMcpPoint = indexFingerPoints[.handLandmarkKeyIndexMCP] else {         self.state = "no index"         return       }               guard let middleTipPoint = middleFingerPoints[.handLandmarkKeyMiddleTIP],          let middleDipPoint = middleFingerPoints[.handLandmarkKeyMiddleDIP],          let middlePipPoint = middleFingerPoints[.handLandmarkKeyMiddlePIP],          let middleMcpPoint = middleFingerPoints[.handLandmarkKeyMiddleMCP] else {         self.state = "no middle"         return       }               guard let ringTipPoint = ringFingerPoints[.handLandmarkKeyRingTIP],          let ringDipPoint = ringFingerPoints[.handLandmarkKeyRingDIP],          let ringPipPoint = ringFingerPoints[.handLandmarkKeyRingPIP],          let ringMcpPoint = ringFingerPoints[.handLandmarkKeyRingMCP] else {         self.state = "no ring"         return       }               guard let littleTipPoint = littleFingerPoints[.handLandmarkKeyLittleTIP],          let littleDipPoint = littleFingerPoints[.handLandmarkKeyLittleDIP],          let littlePipPoint = littleFingerPoints[.handLandmarkKeyLittlePIP],          let littleMcpPoint = littleFingerPoints[.handLandmarkKeyLittleMCP] else {         self.state = "no little"         return       }               guard let wristPoint = wristPoints[.handLandmarkKeyWrist] else {         self.state = "no wrist"         return       } ... } Now every line from thumbPoints onwards results in error, I have fixed the first part (not sure if it is correct or not as it cannot compile) to :         let thumbPoints = try observation.recognizedPoints(forGroupKey: VNHumanHandPoseObservation.JointsGroupName.thumb.rawValue)        let indexFingerPoints = try observation.recognizedPoints(forGroupKey: VNHumanHandPoseObservation.JointsGroupName.indexFinger.rawValue)        let middleFingerPoints = try observation.recognizedPoints(forGroupKey: VNHumanHandPoseObservation.JointsGroupName.middleFinger.rawValue)        let ringFingerPoints = try observation.recognizedPoints(forGroupKey: VNHumanHandPoseObservation.JointsGroupName.ringFinger.rawValue)        let littleFingerPoints = try observation.recognizedPoints(forGroupKey: VNHumanHandPoseObservation.JointsGroupName.littleFinger.rawValue)        let wristPoints = try observation.recognizedPoints(forGroupKey: VNHumanHandPoseObservation.JointsGroupName.littleFinger.rawValue) I tried many different things but just could not get the retrieving individual points to work. Can anyone help on fixing this?

Is there a framework that allows for classic image processing operations in real-time from incoming imagery from the front-facing cameras before they are displayed on the OLED screens? Things like spatial filtering, histogram equalization, and image warping. I saw the documentation for the Vision framework, but it seems to address high-level tasks, like object and recognition. Thank you!

How can I achieve full control over Vision Pro's display and effectively render a 2D graph plot on it? I would appreciate guidance on the necessary steps or code snippets. P.s. As per Apple documentation For a more immersive experience, an app can open a dedicated Full Space where only that app’s content will appear. This still does not fulfill the 'flat bounded 2D' requirement as the Spaces provide an unbounded 3D immersive view.