Accessing and exporting LiDAR point cloud

Hello,

There is no api that will give you access to "raw" LiDAR data.

As mentioned in this WWDC video (https://developer.apple.com/videos/play/wwdc2020/10611/?time=1114), "The colored RGB image from the wide-angle camera and the depth ratings from the LiDAR scanner are fused together using advanced machine learning algorithms to create a dense depth map that is exposed through the API."

The dense depth map is exposed via the sceneDepth api.

If the dense depth map is not suitable for your needs, then you should file an enhancement request using Feedback Assistant.

Could you please give me some tips or code examples on how to work with/access the LiDAR data?

There are two developer samples available that utilize the sceneDepth api:

1. The Visualizing a Point Cloud Using Scene Depth sample that you have already found.

2. The Creating a Fog Effect Using Scene Depth sample

If you have questions that are not answered by either of these samples, or the WWDC video then it may be best if you request technical support for the specific issue you have.

Thanks a lot for your answer!

I have now been able to use the sceneDepth api and also to access the resulting CVPixelBuffer. However, I still cannot access the 3D-points. I also studied the first example again and basically what I would like to access are the 3D-points that are stored in the particlesBuffer. I just do not see how that buffer is filled in the example. Where/How does that happen?

However, I still cannot access the 3D-points. I also studied the first example again and basically what I would like to access are the 3D-points that are stored in the particlesBuffer. I just do not see how that buffer is filled in the example. Where/How does that happen?

The particlesBuffer gets filled in the unprojectVertex function in Shaders.metal.

Keep in mind that these points are just arbitrarily selected on a uniform grid. For your use case, it might make more sense to store the information necessary to unproject in the first place, enabling you to experiment with different grids. You would need to store the following information:

1. The sceneDepth.depthMap

2. The capturedImage

3. The sceneDepth.confidenceMap (if you want it)

4. The cameraIntrinsicsInversed matrix

5. The localToWorld matrix

Thank you very much! I finally managed to do it, your answers have been very helpful!

I am also trying to use the depthmap data, but in a very simple way. I am reading just one horizontal line (portrait orientation) from the Lidar (which is 192 pixels/distances, so to say). The data is somehow unexpected. When the iPad is parallel to a flat wall, say in a distance of 0.3 meter, I would expect to read that 0,3 meter only for the datapoint in the middle of the data line (the one exactly opposite to the camera.) Datapoints at the very left and right ([0] and [191]) should show approx. 0,33 meter. But they do not. All data of that line is 0,3 meter. Once I scan other objects than a wall the data make more sense but still the data seem to be squeezed (a circle column is shown like an ellipse...). Is there an explanation for the data? Or is there a way to get readings that comply to Pythagoras-triangle-theory in that parallel-to-wall experiment? Any insight would be highly appreciated.

Hello :) Did you find a solution for your problem? I am currently working on a similar thing, I am also struggling with providing a point cloud to my neuronal network within swiftUI. I am able to access the pixel buffer, but not the point cloud data. Kind regards ;)