How to attach point cloud(or depth data) to heic?

Question

pseudo-programmer OP

Created Nov ’23

Replies 1

Boosts 2

Participants 2

I'm developing 3D Scanner works on iPad.

I'm using AVCapturePhoto and Photogrammetry Session.

photoCaptureDelegate is like below:

extension PhotoCaptureDelegate: AVCapturePhotoCaptureDelegate {

	func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto photo: AVCapturePhoto, error: Error?) {

		let fileUrl = CameraViewModel.instance.imageDir!.appendingPathComponent("\(PhotoCaptureDelegate.name)\(id).heic")
		let img = CIImage(cvPixelBuffer: photo.pixelBuffer!, options: [ .auxiliaryDepth: true, .properties: photo.metadata ])
		let depthData = photo.depthData!.converting(toDepthDataType: kCVPixelFormatType_DepthFloat32)
		let colorSpace = CGColorSpace(name: CGColorSpace.sRGB)
		let fileData = CIContext().heifRepresentation(of: img, format: .RGBA8, colorSpace: colorSpace!, options: [ .avDepthData: depthData ])
		try? fileData!.write(to: fileUrl, options: .atomic)
	}
}

But, Photogrammetry session spits warning messages:

Sample 0 missing LiDAR point cloud!
Sample 1 missing LiDAR point cloud!
Sample 2 missing LiDAR point cloud!
Sample 3 missing LiDAR point cloud!
Sample 4 missing LiDAR point cloud!
Sample 5 missing LiDAR point cloud!
Sample 6 missing LiDAR point cloud!
Sample 7 missing LiDAR point cloud!
Sample 8 missing LiDAR point cloud!
Sample 9 missing LiDAR point cloud!
Sample 10 missing LiDAR point cloud!

The session creates a usdz 3d model but scale is not correct.

I think the point cloud can help Photogrammetry session to find right scale, but I don't know how to attach point cloud.

Boost

Answer 1

Palace Software OP

3d

Did you ever get an answer to this problem? It's something we have struggled with since the iOS 17 release and it seems to still be a problem with the iOS 18 beta 3. We're using a TIF container for the Depth Map, feeding this into the PhotogrammetrySample, but still we get this error message. We also see incorrect scale in the resulting USDZ models, but the scale isn't consistently too small or too large ... it seems fairly random.

0