Hi, I need your advice about intrinsic matrix issue in iPad 11 inch 3rd generation
I've developed IOS measurement app launched in iPad.
My IOS version is 14.8
Currently, I'm using TrueDepth camera in iPad 11 inch 2nd generation device.
Since iPad 11 inch 2nd generation discontinued, I was about to move to iPad 11 inch 3rd generation and I found some issues in AVCameraCalibrationData.
The test environment is below.
- Measurement object: distance between two vertexs of aruco marker that is square with 2.6cm edge.
- Measurement method: I detected vertexs of aruco marker using OpenCV tool. With depth map and intrinsic matrix from TrueDepth, I calculated world coordinate (x, y, z) of that vertexs. Finally euclidean distance between two vertexs is calculated. Depth map to (x, y, z) conversion code in python is following.
def get_xyz(depth_map, intrinsic_matrix):
intrinsic = np.array(intrinsic_matrix['intrinsic_matrix'])
imat_dim = np.array(intrinsic_matrix['intrinsic_matrix_reference_dimensions'], ndmin=2)
image_dim = np.array([depth_map.shape[1], depth_map.shape[0]], ndmin=2)
ratio = imat_dim / image_dim
intrinsic /= np.hstack([ratio, np.ones([1, 1])])
u = np.arange(depth_map.shape[1]).reshape((1, -1))
v = np.arange(depth_map.shape[0]).reshape((-1, 1))
x = (u - intrinsic[2][0]) * depth_map / intrinsic[0][0]
y = (v - intrinsic[2][1]) * depth_map / intrinsic[1][1]
xyz = np.stack([x, y, depth_map], axis=2)
return xyz
I'm struggling in two problems.
- There's problem in intrinsic matrix.
iPad 2nd gen can measure distance between two vertexs correctly (2.6cm) but 3rd gen is giving small value (1.8cm)
I checked RGB and depth map are synced in 3rd gen.
I found that 2nd gen and 3rd gen give different intrinsic matrix.
2nd gen intrinsic matrix
3rd gen intrinsic matrix
It's weird in that principal point of 3rd gen intrinsic matrix is highly skewed to left-bottom area and focal length is much bigger than that of 2nd gen.
- lens distortion parameter malfunction
3rd gen lens distortion parameter also is malfunctioning. I'm using re-implementation code of [https://frost-lee.github.io/rgbd-iphone/] in python to get rectilinear image and 3rd gen gave even more distorted result.
2nd gen result
3rd gen result
3rd gen lens distortion center (814.24, 1860.73) is also skewed to left-bottom area than that of 2nd gen (1156.00, 1559.13)
Can a iPad device fails to factory camera calibration? Or is there any misconception to me?
Is there anyone who suffered from same problem?
I appreciate for your sharing.
Firstly, I correct misleading information above. I used iPad pro 11 inch 2nd gen, 3rd gen for test, not iPad 11 inch.
Following the previous test, I tried to calibrate TrueDepth camera on my own. I used 14 checkerboard images and OpenCV camera calibration functions. A sample checkerboard image is below.
Calibration steps and codes are referred to [https://docs.opencv.org/4.x/dc/dbb/tutorial_py_calibration.html].
Intrinsic matrix I found is below and it works perfectly on marker distance measurements.
I guess extrinsic matrix is might needed to compensate weird original intrinsic matrix though extrinsic matrix is currently useless as it's given as identity matrix.
+) Even if I assumed the intrinsic matrix is for ultra wide field-of-view, measurement was still wrong.