I would like to extract depth data for a given point in ARSession.currentFrame.smoothedSceneDepth
.
Optimally this would end up looking something like:
ARView.depth(at point: CGPoint)
With the point being in UIKit coordinates just like the points passed to the raycasting methods.
I ultimately to use this depth data to convert a 2D normalized landmark from a Vision image request into a 3D world space coordinate in the 3D scene - I only lack the accurate depth data for a given 2D point.
What I have available is:
- The normalized landmark from the Vision request.
- Ability to convert this^ to AVFoundation coordinates.
- Ability to convert this^ to screen-space/display coordinates.
When the depth data is provided correctly I can combine the 2D position in UIKit/screen-space coordinates with the depth (in meters) to produce an accurate 3D world position with the use of ARView.ray(through:)
What I have not been able to figure out is how to get this depth value for this coordinate on screen.
I can index the pixel buffer like this:
extension CVPixelBuffer {
func value(for point: CGPoint) -> Float32 {
CVPixelBufferLockBaseAddress(self, .readOnly)
let width = CVPixelBufferGetWidth(self)
let height = CVPixelBufferGetHeight(self)
//Something potentially going wrong here.
let pixelX: Int = width * point.x
let pixelY: Int = height * point.y
let bytesPerRow = CVPixelBufferGetBytesPerRow(self)
let baseAddress = CVPixelBufferGetBaseAddress(self)!
assert(kCVPixelFormatType_DepthFloat32 == CVPixelBufferGetPixelFormatType(depthDataMap))
let rowData = baseAddress + pixelY * bytesPerRow
let distanceAtXYPoint = rowData.assumingMemoryBound(to: Float32.self)[pixelX]
CVPixelBufferUnlockBaseAddress(self, .readOnly)
return distanceAtXYPoint
}
}
And then try to use this method like so:
guard let depthMap = (currentFrame.smoothedSceneDepth ?? currentFrame.sceneDepth)?.depthMap else { return nil }
//The depth at this coordinate, in meters.
let depthValue = depthMap.value(for: myGivenPoint)
The frame semantics [.smoothedSceneDepth, .sceneDepth]
have been set properly on my ARConfiguration. The depth data is available.
If I hard-code the width and height values like so:
let pixelX: Int = width / 2
let pixelY: Int = height / 2
I get the correct depth value for the center of the screen. I have only been testing in portrait mode.
But I do not know how to index the depth data for any given point.