Arkit 4 Lidar intrinsics & extrinsics

Question

Created Oct ’20

Replies 3

Boosts 0

Views 2.9k

Participants 4

Hello,
I am trying to retreive the intrinsics & extrinsics of the lidar wrt RGB on an Ipad Pro 2020.
SDK does not seem to give them.
What should I use ?

I've tried :
https://developer.apple.com/documentation/arkit/arcamera/2875730-intrinsics?language=objc

both in swift and objective-c and only the RGB seem to contain data. Using the following callbacks :

ObjC :

Code Block     simd_float3x3 mat = self.sceneView.session.currentFrame.camera.intrinsics;
    simd_float3x3 matDepth = self.sceneView.session.currentFrame.capturedDepthData.cameraCalibrationData.intrinsicMatrix ;
    float fx = mat.columns[0].x;
    float fy = mat.columns[1].y;
    float cx = mat.columns[2].x;
    float cy = mat.columns[2].y;
    float teta = mat.columns[1].x;
    NSLog(@" fx %f", fx);
    NSLog(@" fy %f", fy);
    NSLog(@" cx %f", cx);
    NSLog(@" cy %f", cy);
    NSLog(@" teta %f", teta);
    float fx2 = matDepth.columns[0].x;
    float fy2 = matDepth.columns[1].y;
    float cx2 = matDepth.columns[2].x;
    float cy2 = matDepth.columns[2].y;
    float teta2 = matDepth.columns[1].x;
    NSLog(@" fx2 %f", fx2);
    NSLog(@" fy2 %f", fy2);
    NSLog(@" cx2 %f", cx2);
    NSLog(@" cy2 %f", cy2);
    NSLog(@" teta2 %f", teta2);

Results in :
fx 1619.127563
fy 1619.127563
cx 934.813416
cy 713.917847
teta 0.000000
fx2 0.000000
fy2 0.000000
cx2 0.000000
cy2 0.000000
teta2 0.000000

In Swift :

Code Block      print ("RGB:", self.session.currentFrame.camera.intrinsics.debugDescription)
    }
print ("Depth : " ,  self.session.currentFrame.capturedDepthData.debugDescription)
    }

results in :
RGB: simd_float3x3([[1598.34, 0.0, 0.0], [0.0, 1598.34, 0.0], [935.70917, 713.61804, 1.0]])
Depth : nil

Bonus Question :
Is there any sample project code where RGB and Lidar are aligned ?

Boost

Answer 1

Vision Pro Engineer OP

Apple

Oct ’20

capturedDepthData contains depth data for experiences using the front-facing camera.
Try adding .sceneDepth to your configuration's frame semantics, then look at the sceneDepth property on ARFrame: https://developer.apple.com/documentation/arkit/arframe/3566299-scenedepth.

You can check out a developer sample that visualizes a colored point cloud based on RGB and depth data here:
https://developer.apple.com/documentation/arkit/visualizing_a_point_cloud_using_scene_depth

0

Answer 2

xsmarty OP

Oct ’20

Hi,

The mentioned fields do not seem to offer anything else than the raw data.
Following your answer:

Code Block print ("Depth: ",  frame.sceneDepth?.depthMap) 
print ("confidenceMap: ",  frame.sceneDepth?.confidenceMap.debugDescription)

returns

Code Block Depth:  Optional(<CVPixelBuffer 0x2829ea580 width=256 height=192 bytesPerRow=1024 pixelFormat=fdep iosurface=0x281ac8dc0 attributes={
    Height = 192;
    IOSurfaceProperties =     {
        IOSurfacePurgeWhenNotInUse = 1;
    };
    PixelFormatType = 1717855600;
    PlaneAlignment = 16;
    Width = 256;
} propagatedAttachments={
} nonPropagatedAttachments={
}>)
confidenceMap:  Optional("Optional(<CVPixelBuffer 0x2829ea6c0 width=256 height=192 bytesPerRow=256 pixelFormat=L008 iosurface=0x281ac8e10 attributes={\n    Height = 192;\n    IOSurfaceProperties =     {\n        IOSurfacePurgeWhenNotInUse = 1;\n    };\n    PixelFormatType = 1278226488;\n    PlaneAlignment = 16;\n    Width = 256;\n} propagatedAttachments={\n} nonPropagatedAttachments={\n}>)")

No trace of any intrinsics or extrinsics.

Having looked at the sample https://developer.apple.com/documentation/arkit/visualizing_a_point_cloud_using_scene_depth,
It is my understanding that in the file Shaders.metal; the RGB and Depthmap are sampled at the same relative pixel (via linear interpolation), and then that the intrinsics of the RGB are used (the projection matrix). This equals to adding a depth for each u,v, coordinate in RGB and then projecting using the RGB sensor.

This approach unfortunately does not yield perfect alignment between lidar and rgb since fov, distortions and sensors are not the same/aligned.
Therefore is there any way to retrieve The Projection Matrix K, the Extrinsic Matrix [Rt] ?

Projection Matrix RGB = K'*[R' t']; => This we have through ARFrame.camera.intrinsics (camera/image/world)
Projection Matrix Lidar= K"*[R'' t''']; => This is missing.

Best.
X.

0

Answer 3

Bookhahn OP

Nov ’20

Hi,

color image and depth image are already aligned. That means the intrinsics of the Lidar are only scaled in relation to the color camera. As the depth image has a resolution of 584 x 384 (frame.sceneDepth!.depthMap) and the color image 3840 x 2880, you get fxD, fyD, cxD and cyD as follows:

fxD = 534/3840 * 1598.34
fyD = 384/2880 * 1598.34
cxD = 534/3840 * 935.70917
cyD = 384/2880 * 713.61804

Before transforming the pointcloud to world coordinates, you have to flip them around the X axis to OpenGL coordinate system.

Best Regards!

0