n ARKit viewMatrix is define as the transform matrix from world coordinate frame to camera frame coordinate. eg- this is from the console of Xcode and it is the output of frame?.camera.viewMatrix(for: orientation) viewMatrix is Optional(simd_float4x4([[-0.21701495, -0.8391591, -0.49871516, 0.0], [-0.9272899, 0.017577814, 0.3739312, 0.0], [-0.3050214, 0.54360217, -0.78195834, 0.0], [0.15720224, -0.23272672, 0.0491977, 1.0]])) forth raw is not [0,0,0,1] like. could you please explain me why is this?

Was looking for somewhat consistent Xcode AR support if you could hopefully point me in the right direction? Insta @NanoArtsNet

I am doing an image tracking Reality Composer experience. Currently, the ground plane in Reality Composer clips the object as it intersects the ground. Is there a way to turn that "feature" off? or to be able to raise the image tracker to be above the ground instead of on the ground? https://www.dropbox.com/s/hbahcpcr4rmnl5p/screenCapture.jpg?dl=0 I would also very much like to be able to turn off people and space occlusion on usdz and .reality files, but I can't seem to do that. thanks for any ideas/help. Dale

I'd like to recreate the apple event invitation AR for my hobby,I wonder how the sphere behind AR disappear at beginning.At first I thought the Apple logo was a static image,but I noticed that the image behind the logo moved with the camera of view.

Hello Everyone, I would like to use apple native framework(ARKit, RealityKit) and Lidar sensor( available in iPhone 12, 13 Pro, Pro max) to scan particular object in 3D and export that 3D model into OBJ formate. Reference link added below, Thanks in advance. https://www.youtube.com/watch?v=wfPYLAnMQt4

We're going to need a way to disable occlusion in AR quicklook; it would literally expand the amount of usecases by a factor 10. Anything that is bigger than 5x5 gets occluded now; for many of our clients, this doesn't work. Please add the ability to add a flag that disables occlusion.

In preview, the 3D model is visible in the background but once the camera button is pressed the image in the screenshot is just the 3D model with a black background which is also saved. QLPreviewController implementation: `func collectionView(_ collectionView: UICollectionView, didSelectItemAt indexPath: IndexPath) { thumbnailIndex = indexPath.item let previewController = QLPreviewController() previewController.dataSource = self previewController.delegate = self present(previewController, animated: true) } func numberOfPreviewItems(in controller: QLPreviewController) -> Int { return 1 } func previewController(_ controller: QLPreviewController, previewItemAt index: Int) -> QLPreviewItem { let url = Bundle.main.url(forResource: models[thumbnailIndex], withExtension: "usdz")! return url as QLPreviewItem }` console output is: 2021-10-07 21:26:30.567439+0200 CPARTest[522:28543] Writing analzed variants. 2021-10-07 21:26:30.620474+0200 CPARTest[522:28738] Metal GPU Frame Capture Enabled 2021-10-07 21:26:30.621447+0200 CPARTest[522:28543] Writing analzed variants. 2021-10-07 21:26:30.623724+0200 CPARTest[522:28738] Metal API Validation Enabled <SCNNode: 0x280e04d00 | no child> -> <ARImageAnchor: 0x2809085b0 identifier="64834AA5-7912-1102-CCC2-77992D9D8FE0" transform=<translation=(0.239411 -0.028021 0.027675) rotation=(84.68° -98.90° -2.02°)> referenceImage=<ARReferenceImage: 0x281600a80 name="cake" physicalSize=(0.050, 0.050)> tracked=YES> 2021-10-07 21:26:34.972808+0200 CPARTest[522:28772] [Session] ARSession <0x104c1ff70>: ARSessionDelegate is retaining 11 ARFrames. This can lead to future camera frames being dropped. 2021-10-07 21:26:34.989003+0200 CPARTest[522:28741] [Session] ARSession <0x104c1ff70>: ARSessionDelegate is retaining 12 ARFrames. This can lead to future camera frames being dropped. 2021-10-07 21:26:35.006186+0200 CPARTest[522:28772] [Session] ARSession <0x104c1ff70>: ARSessionDelegate is retaining 13 ARFrames. This can lead to future camera frames being dropped. 2021-10-07 21:26:35.023000+0200 CPARTest[522:28750] [Session] ARSession <0x104c1ff70>: ARSessionDelegate is retaining 14 ARFrames. This can lead to future camera frames being dropped. 2021-10-07 21:26:35.039644+0200 CPARTest[522:28739] [Session] ARSession <0x104c1ff70>: ARSessionDelegate is retaining 15 ARFrames. This can lead to future camera frames being dropped. 2021-10-07 21:26:35.537424+0200 CPARTest[522:28741] [default] [self.extensionContext conformsToProtocol:auxHostProtocol.protocol] - /Library/Caches/com.apple.xbs/Sources/ExtensionFoundation/ExtensionKit-49.1/ExtensionFoundation/Source/NSExtension/NSExtensionSupport/EXExtensionContext.m:332: Class QLPreviewExtensionHostContext does not conform to aux host protocol: QLRemotePreviewHost

I have set maximumNumberOfTrackedImages = 100, and ARKit5 can recognize more than 4 images but not simultaneously. ARKit5 is like ARKit4 only tracking up to 4 images at once. How can I make it track more images at once? Thank you!

Hi everyone , just started to have a look about XCode and SWIFT unite and after creating my struct ContentView: view i wanted to put a BTN that start he AR experience (with the function to scan an element from a voucher) Btw i didn't manage to start the camera opening with the press of the Btn. Knowing that this is a very basic question i ll ty in advice who is gonna spend some time for an ansewr.

Possibly the wrong forum to ask the question, but I figured it would be the best place to start if it is. I am interested in capturing distance dimensions of a horse using my camera in real time. I have attached an image that roughly shows what data I am interested in capturing. I am aware that I am asking two things to occur namely... 1 - the automated placement of the markerless anatomical features on the horse. There are packages like DeepLabCut available, but I am unsure if they can cope with the variation in distance from the camera to the horse that is most likely to occur, thus making the second part of the project harder to complete (unless I am mistaken, which is possible) 2 - While the horse is standing relatively still I was hoping that I could get the distance from me to Points A and F (see image attached) so that I could optimally place the camera between the two points so as to properly reflect the horses dimensions. Once I was happy with how the horse was standing and my distance from the horses shoulder and hindleg, I would press a button to capture the dimensions of the horse and make some calculations (eg, distance from C to F via D) that I would store in a database of some description. I was just wondering if this was possible and best handled with ARKit? It seems that capturing the distance from the camera to the horse is possible with LiDAR or TrueDepth? Any advice appreciated.

ARSCNView and AVCaptureVideoPreviewLayer can use at the same time? I want to use AVCaptureVideoPreviewLayer to render my camera data . but ,I have another sdk which use ARSCNView to render camera data. so , how can I use bother of them

I want to capture the RGB frame and depth map of frame on button click. The RGB frame is sent to an image classifier and the output is spoken on audio channel. I'm able to capture the RGB data using AVCaptureSession and it works well as intended. However, when I configure and run the ARSession, the app freezes on the first frame, (preview of RGB camera). It captures and saves the depth map but the RGB image isn't captured anymore.     @Published var isTaken = false     @Published var session = AVCaptureSession()     @Published var ARsession = ARSession()     @Published var alert = false     @Published var output = AVCapturePhotoOutput()     @Published var depthMap: [Float32] = []     // preview layer of camera     @Published var preview: AVCaptureVideoPreviewLayer!     @Published var isSaved = false     // to save the captured image     @Published var picData = Data(count: 0)          var classifier = Classifier()     var speachSynthesizer = SpeechSynthesizer()          // handle permissions     func check(){         switch AVCaptureDevice.authorizationStatus(for: .video){                      case .notDetermined:             // ask f or permission             AVCaptureDevice.requestAccess(for: .video) { status in                 if status{                     self.setup()                 }             }         case .denied:             self.alert.toggle()             return         case .authorized:             //setup session             setup()             ARSetup()                          return         @unknown default:             return         }     }          // setup session     func setup(){                  do{                          let device = AVCaptureDevice.default(.builtInDualCamera, for: .video, position: .back)             self.session.beginConfiguration()             // change .builtinDualCamera to microphone for audio, for: can be changed for depth data                          let input = try AVCaptureDeviceInput(device: device!)                          if self.session.canAddInput(input){                 self.session.addInput(input)             }                          if self.session.canAddOutput(output){                 self.session.addOutput(output)             }                          self.session.commitConfiguration()                     }catch{             print(error.localizedDescription)         }                       }          func ARSetup(){         self.ARsession.delegate = self         let configuration = ARWorldTrackingConfiguration()         if ARWorldTrackingConfiguration.supportsFrameSemantics(.sceneDepth){             configuration.frameSemantics = .sceneDepth             print("AR configured...")         }         self.ARsession.run(configuration)          }          func takePic(){         DispatchQueue.global(qos: .background).async {             self.output.capturePhoto(with: AVCapturePhotoSettings(), delegate: self)             self.captureDepth()             //self.session.stopRunning()         }     }          func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto photo: AVCapturePhoto, error: Error?) {         if error != nil{             return         }                  print("Picture Taken....")                 guard let imageData = photo.fileDataRepresentation() else {return}                  self.picData = imageData         predict()                       }               func predict(){         let ciImage = CIImage(data: self.picData)         classifier.detect(ciImage: ciImage!)         print(classifier.result)         self.speachSynthesizer.speak(text: classifier.result!)                  //uploadImage(data: self.picData, filename: classifier.result!)              }                         func captureDepth(){         if let currentFrame = self.ARsession.currentFrame{             print("depth captured...")             guard let depthData = currentFrame.sceneDepth?.depthMap else {return}             self.saveDepth(depthData: depthData)         }     }          func saveDepth(depthData: CVPixelBuffer){         let width = CVPixelBufferGetWidth(depthData)         let height = CVPixelBufferGetHeight(depthData)         //let depthSize = CGSize(width: width, height: height)                  let ciImage = CIImage(cvPixelBuffer: depthData)         let context = CIContext.init(options: nil)         guard let cgImageRef = context.createCGImage(ciImage, from: CGRect(x: 0, y: 0, width: width, height: height)) else {return}         let uiImage = UIImage(cgImage: cgImageRef)         print("depth saved...")         UIImageWriteToSavedPhotosAlbum(uiImage, nil, nil, nil)     }                    }

I need to save certain frames' capturedImage to heic file but also include proper exif metadata. Right now when I save the file there is no exif data defined. How do I go about adding this? Looking at Apple's CaptureSample app I can see the exif metadata is included in the images captured but isn't through arkit. Is there a way to get exif data while arkit is running? Thank you!

We are developing an augmented reality (AR) game for iOS. In the game scene, we need to use both the iPhone Rear Camera and the Front Camera. In the case of Augmented Reality with the Rear Camera, we need to use the Front Camera to make video calls at the same time. In the process of application software development, we use the Rear Camera as ARCamera, and through the configuration ARWorldTrackingConfiguration class to create a ARSession instance, then use ARSCNView object to build the AR view shows, Augmented Reality with the Rear Camera we've done. But there was a problem. We couldn't get the video stream from the Front Camera at the same time. The question is: How do we make video calls with a Front Camera while having a Augmented Reality with the Rear Camera?We want the details of implementation.

Hello developers, I am recently struggling with providing the right data for a deep learning model i want to integrate into my swift app. (I am fairly new to swift and ios dev, please bear with me.) The App is supposed to run on an iPad Pro with Lidar sensor, no other devices. I'm working with Dense Fusion 6DoF pose estimation , which requires an RGB-D Image as input. (it's trained on the ycb-video dataset) I already looked up different examples on how i can stream depth data from the camera (fog example) and also how to capture an Image with depth information. So I started a session that gets video and depth data from the CVPixelBuffer as input. However, I don't really know what to do with them after that and how to fuse them together to create one RGBD image. Also I want to use the predicted pose to place a 3d model into a scene so i can append AR content to it. (The apps purpose is to check whether Deep Learning can outperform RealityKit in things like poor lighting conditions, etc) I'd be glad about every little help. Thanks in advance!

We need to develop an iOS app that will make heavy use of ARSessions with ARFaceTrackingConfiguration, in more or less 100% of the screens. Our main concern is related to battery usage. We're in discovery phase, so I'm trying to find/research all possible risks related to it. For now, what I found is: ARKit sessions don't "drain" the battery (I did some testing with my phone), they just consume more that common apps (more or less the same amount of battery than having the iPhone camera open for a long period of time, with the addition that ARSession is continually creating frames in order to track the face -real time video processing). I'm aware of run(_:options:) and pause() methods on ARSession, in order to pause the tracking functionality when it's not necessary. I'm aware about lowPowerModeEnabled and we can react to changes in that property. I searched Apple Developer for some official article that provides general considerations about battery usage in an ARKit session, but I couldn't find anything dedicated to the topic. I'm wondering if there is a really important thing that I should be aware and I'm missing when implementing the feature. It's the first time I'll work with ARKit and it's critical for the project. Thanks in advance!

I am trying to use an image from ARFrame.capturedImage during ARSession with ARWorldTrackingConfiguaration for computer vision task. Is this image is corrected for distortion errors (such as strait lines in object space remain strait on the image ) or I have to apply some sort of correction myself?

I want to make a web-based augmented reality app on iOS devices, but can I place multiple 3D models in different locations in a single link, and how do I place them