Not with Object Capture Session, I tried with RealityKit

With code below, I added color and depth image from RealityKit ARView, and ran Photogrammetry on iOS device, the mesh looks fine, but the scale of the mesh is quit different with real world scale.

let color = arView.session.currentFrame!.capturedImage let depth = arView.session.currentFrame!.sceneDepth!.depthMap

    //😀 Color
    let colorCIImage = CIImage(cvPixelBuffer: color)
    let colorUIImage = UIImage(ciImage: colorCIImage)
    let depthCIImage = CIImage(cvPixelBuffer: depth)
    let heicData = colorUIImage.heicData()!

    
    let fileURL = imageDirectory!.appendingPathComponent("\(scanCount).heic")
    do {
        try heicData.write(to: fileURL)
        print("Successfully wrote image to \(fileURL)")
    } catch {
        print("Failed to write image to \(fileURL): \(error)")
    }
    
    //😀 Depth
    let context = CIContext()
    let colorSpace = CGColorSpace(name: CGColorSpace.linearGray)!        
    let depthData = context.tiffRepresentation(of: depthCIImage,
                                               format: .Lf,
                                               colorSpace: colorSpace,
                                               options: [.disparityImage: depthCIImage])
    let depth_dir = imageDirectory!.appendingPathComponent("IMG_\(scanCount)_depth.TIF")
    try! depthData!.write(to: depth_dir, options: [.atomic])
    print("depth saved")

And also tried this.

    let colorSpace = CGColorSpace(name: CGColorSpace.linearGray)
    let depthCIImage = CIImage(cvImageBuffer: depth,
                               options: [.auxiliaryDepth : true])
    let context = CIContext()
    
    let linearColorSpace = CGColorSpace(name: CGColorSpace.linearSRGB)
    guard let heicData = context.heifRepresentation(of: colorCIImage,
                                                    format: .RGBA16,
                                                    colorSpace: linearColorSpace!,
                                                    options: [.depthImage : depthCIImage]) else {
        print("Failed to convert combined image into HEIC format")
        return
    }

Does Anyone know why and how to fix this?

Replies

Hi,

I built mine with arkit. Interestingly my depth is correct And scale is correct, but my orientation is wrong. I tried feeding in gravity data via makerapple dictionary using “8” . That is where Apple secretly stores its metadata for gravity. I also tried writting in metadata for intrinsics and extrinsics but I don’t know if object capture is actually using it.

regards,

peter