Raycasting VNFaceLandmarkRegion2D

Does it make sense to raycast face landmarks in the first place? Assuming face is stationary long enough?

Hello @jimbang,

More specifically, how to construct the "from" parameter of the frame's raycastQuery from a VNFaceLandmarkRegion2D point?

I assume you are referring to ARFrame's raycastQuery method?

The point parameter is documented in ARFrame.h as:

/*
@param point A point in the image-space coordinate system of the captured image.
 Values should range from (0,0) - upper left corner to (1,1) - lower right corner.
*/

Do the points need to be flipped vertically? Is there any other transformation that needs to be performed on the points prior to passing them to raycastQuery?

Vision always returns its observations in normalized image-space. Whether a flip is necessary here or not depends on whether the orientation you processed the image with in Vision matches the orientation of the frame.

Does it make sense to raycast face landmarks in the first place? Assuming face is stationary long enough?

Not really, ray-casting is for detecting intersections with real-world surfaces. I think if you described the feature your trying to build at a higher level I could give a recommendation :)

--Greg

Hello Greg,

Yes, I’m referring to ARFrame's raycastQuery.

Essentially, I’d like to measure human facial features using the rear camera with LiDAR, and take advantage of any temporal refinement available in either ARKit or AVFoundation.

Is depth data refined over time during an ARKit session?

I’d really appreciate any recommendations—especially from a DTS engineer.

Hello @jimbang,

I see, is there a specific reason that you want to do this with the rear camera rather than the front? The front camera will enable you to take advantage of ARFaceAnchor.

I don't have a strong recommendation for doing this with the rear camera. You can use Vision to detect 2D face landmarks, and then use depth data to get world points for those landmarks, but the measurements from that approach are likely to not meet your requirements.

--Greg