This is none of our developer's business. It is Apple's job to solve this problem. We only use the API provided by Apple.

Goal: Multiple users in a REAL 3D space, e.g. conference/operation room, share spatial experiences using AR. Given: The real concrete 3D space is given. Real-time RGB depth data is collected from individual user sensors. Additionally, real-time accelerometer and magnetic sensor data. The real objects (floor, wall, ceiling, table, ...) are motionless. Users move in the space. Virtual objects/content are moved by users. Users, real/virtual objects/content interact with each other. Solutions: The real indoor space must be shared by users and serve as reference data. Floors, walls, ceilings, columns, tables, etc. serve as permanent environmental anchors. Users and virtual objects/content are moving.

At the idea level, could a solution like the following be developed? Examples of indoor physical objects such as floors, walls, ceilings, columns, and tables can act as a kind of environmental anchor. For example, a single wall (plane) can only provide the normal and a point on it (incomplete 6DoF anchor). But 3 non-parallel planes as a set can provide a complete 6DoF anchor. Would it be possible to calculate the 6DoF of individual devices using such environmental anchors as intermediaries? Gravitational and magnetic directions would be useful information. YouTube video: Hv2uA6k8Oig

Anchor is a representation of 3D position and orientation (6DoF). For example, object surface meshes, device, hands, tables, floors, walls, etc. have their own 6DoF anchors. They are represented as MeshAnchor, DeviceAnchor, HandAnchor, PlaneAnchor, ... by ARKit in iOS/visionOS. 3D position and orientation (6DoF) is mathematically described by a 4x4 matrix. Understanding the “rotation matrix R” is the core. Vector & matrix, and linear algebra are the foundations of “spatial computing”.

ARKit generates and processes LiDAR 3D data in 3 phases (data formats): 576 (= 4 x 16 x 3 x 3) raw distance points at very high frequency (2,000 Hz?): 4 VCSEL (Vertical-Cavity Surface Emitting Laser), 16 stacks, 3 x 3 DOE (Diffractive Optical Elements). DepthMap (256 x 192) at 60 Hz. MeshAnchor at about 1 Hz. Apple provides the ARKit API to app developers: iOS/iPadOS: DepthMap and MeshAnchor. visionOS: MeshAnchor. The undisclosed ARKit API for 576 raw distance points will open up new applications. Object box, plane, sphere, cylinder, cone and torus can be detected and measured at a speed of max. 1,000 Hz on Vision Pro: https://github.com/CurvSurf/FindSurface-RealityKit-visionOS-Real-Time Any content/materials/images/videos can be drawn on the object surfaces: YouTube video, Hv2uA6k8Oig Is there any way to get the access API to 576 raw distance points from LIDAR?

The videos were recorded about 10 years ago with Intel RealSense 3D cameras. We measure a box from the inside or outside. The videos show examples of external observations. Room measurement is internal observation. OpenCV has nothing to do with the videos. Real-time box measurement requires many problems to be considered and solved. Not an easy task. If allowed, in a few months we may release the source code of a real-time box measurement app for iOS that uses our proprietary runtime library “FindBox” that processes Apple LiDAR 3D point clouds.

Box measurement (size and 6DoF) is an unsolved industrial application problem in warehouse and airport automation. Once there is a solution, it must be great. It could be solved by processing both 3D point cloud and 2D image. Following videos show partial solutions processing only point cloud. YouTube Videos: Real-Time Box Measurement, l-i2E7aZY6A Measuring & Tracking of Boxes in Real-Time, mEHaR2mjm9c

Hi, You are right. Fortunately, visionOS offers alternatives: device tracking and hand tracking. Others can see where you are looking, so there are no privacy issues. A little tip: the center of the screen where the device is pointing is inconvenient for you to see. It is better if you set an angle of 10-15 degrees downwards to the device direction.

The LowLevelMesh might be the solution to your problem. You have to manage the meshes yourself. This means you cannot use the boxes or balls provided by RealityKit.

Processing 3D point clouds is not a simple task, it is complicated and not yet fully understood. Recognizing and accurately measuring shape, size and 6DoF in real time is the “Holly Gray” of 3D data processing.

The common algorithms for determining the 6DoF of an object plane are Hough transform (1961) and RANSAC (Random Sample Consensus, 1981). They use the plane equation “ax + by + cz + d = 0 with ||(a, b, c)|| =1”. (a, b, c) is the normal of the plane, d is the distance of the plane from the origin. One of the serious weaknesses is that the parameter “d” has practically nothing to do with the position of the object plane. It is a completely unknown reality. An object plane must be defined with normal and an interior point on the plane. The best position is the centroid of the measurement points. The size (i.e. the boundary) of the object plane can be determined by the measurement points. The Hough transform and RANSAC algorithms cannot realize this. There must be a new solution algorithm. January 14, 2025.

A real object surface has shape, size, and 6DoF (position & rotation). Once we developers can be getting such information, we can do much in AR/VR/MR/XR, spatial computing, robotics, digital twin, reverse engineering, CAD/CAM, etc. What visionOS overlooked is that the position of a real object must be defined inside the object. visionOS places object (mesh, plane, etc.) anchors somewhere about the app origin.

This may be helpful. ARKit Point Cloud Recorder.

This may be helpful. VideoMaterial Advertisement

The source code of the FindSurfaceAD app is now accessible on GitHub CurvSurf. Photos/videos are imported from Photos app.