Geometry recognition and measurement from MeshAnchor

Spatial Computing ARKit
JoonAhn OP
Created Jun ’24
Replies 8
Boosts 0
Views 920
Participants 1

FYI.

The source code of the FindSurface demo app for Apple Vision Pro (visionOS) is available now. The Swift package of FindSurface™ library is required to build the source code into the demo app. https://github.com/CurvSurf/FindSurface-visionOS

After starting the app, the floating panels (below) will appear on your right side, and you will see wireframe meshes that approximately describe your environments. Performing a spatial tap (pinching with your thumb and index finger) with staring at a location on the meshes will invoke FindSurface, with an indicator (blue disk) appearing on the surface you've gazed.

Voice commands:

  • “Tap” – Spatial tap (gazing & pinching). Invoke FindSurface.
  • “Tap plane” – Plane selection.
  • “Tap sphere” or “Tap ball” – Sphere selection.
  • “Tap cylinder” – Cylinder selection.
  • “Tap cone” – Cone selection.
  • “Tap torus” or “Tap donut” – Torus selection.
  • “Tap accuracy” or “Tap measurement accuracy” – Accuracy selection.
  • “Tap mean distance”, “Tap average distance”, or “Tap distance” – Avg. Distance selection.
  • “Tap touch radius” or “Tap seed radius” – Touch Radius selection.
  • “Tap Inlier” – “Show inlier points” toggle.
  • “Tap outline” – “Show geometry outline” toggle.
  • “Tap clear” – “Clear Scene” click.
Answered by JoonAhn in 807445022

There was a delay in rendering the mesh triangle in the line of sight (ray-casting), so we tried to use the "LowLevelMesh" of visionOS 2.0. The problem is solved. We plan to apply "LowLevelMesh" to all mesh rendering, such as view-triangle, MeshAnchors, and object surface meshes.

Current speed of object surface extraction:

  • Plane: 400 Hz
  • Sphere/cylinder/cone/torus: 200 Hz.

The source code of the app is available: https://github.com/CurvSurf/FindSurface-RealityKit-visionOS-Response-to-Request

Replies  8
Boosts  0
Views  920
Participants  1
JoonAhn OP
Jun ’24

Voice commands:

  • “Tap” – Spatial tap (gazing & pinching). Invoke FindSurface.
  • “Tap plane” – Plane selection.
  • “Tap sphere” or “Tap ball” – Sphere selection.
  • “Tap cylinder” – Cylinder selection.
  • “Tap cone” – Cone selection.
  • “Tap torus” or “Tap donut” – Torus selection.
  • “Tap accuracy” or “Tap measurement accuracy” – Accuracy selection.
  • “Tap mean distance”, “Tap average distance”, or “Tap distance” – Avg. Distance selection.
  • “Tap touch radius” or “Tap seed radius” – Touch Radius selection.
  • “Tap Inlier” – “Show inlier points” toggle.
  • “Tap outline” – “Show geometry outline” toggle.
  • “Tap clear” – “Clear Scene” click.
0
JoonAhn OP
Jul ’24

https://github.com/CurvSurf/FindSurface-RealityKit-visionOS

Once object measurement points (point cloud, mesh vertices, etc.) are collected, the measurement accuracy and average distance between points are fixed.

CurvSurf FindSurface works based on those values ​​and requires additional parameter values ​​for region growing to enlarge the length/width and increase the radius of the estimated object geometry.

After a series of experiments, CurvSurf found the optimal parameter value set for MeshAnchor generated by Apple Vision Pro:

  • Accuracy: 1.5 cm,
  • Average Distance: 10 cm,
  • Lat. Ext.: 10,
  • Rad. Exp.: 5,
  • Touch Radius: 1/4 - 1/2 of object diameter/width [cm].

As an Apple Vision Pro user, you need to adjust the Touch Radius proportional to the approximate diameter or width of your object. It is a problem of object scaling for a range of 1 - 20 meters.

If you like to get a relatively small plane or short cylinder/cone, you set the Lat. Ext. to less than 5.

0
JoonAhn OP
Jul ’24

The FindSurface demo app for Apple Vision Pro (visionOS) is now stabilized: https://github.com/CurvSurf/FindSurface-RealityKit-visionOS

  • Cone2Cylinder, Torus2Cylinder, and Torus2Sphere conversions are implemented. For example, if you try to get a cone from a real cylindrical object (with a diameter of larger than 1 meter), FindSurface will convert the resulting cone to a cylinder.
  • Cones and tori are trimmed to the boundaries of the inlier points.
0
JoonAhn OP
Aug ’24

Real-time preview for object geometry detection and measurement from MeshAnchors. Object planes, spheres, cylinders, cones, and tori can be detected and measured in real time up to 90 Hz. The source code of the app will be available on GitHub CurvSurf.

FindSurface Real-Time Preview - Apple Vision Pro https://youtu.be/CGjhfKxjpUU

0
JoonAhn OP
Aug ’24

The source code of the AVP app will be available on GitHub CurvSurf in the week of Sept. 2, 2024.

FindSurface Real-Time 1 - Apple Vision Pro

https://youtu.be/2aSMBrPTEtg

0
JoonAhn OP
Oct ’24
Accepted Answer

There was a delay in rendering the mesh triangle in the line of sight (ray-casting), so we tried to use the "LowLevelMesh" of visionOS 2.0. The problem is solved. We plan to apply "LowLevelMesh" to all mesh rendering, such as view-triangle, MeshAnchors, and object surface meshes.

Current speed of object surface extraction:

  • Plane: 400 Hz
  • Sphere/cylinder/cone/torus: 200 Hz.

The source code of the app is available: https://github.com/CurvSurf/FindSurface-RealityKit-visionOS-Response-to-Request

0
JoonAhn OP
3w

The source code of real-time app is available: https://github.com/CurvSurf/FindSurface-RealityKit-visionOS-Real-Time

This is a minimized and optimized version of https://github.com/CurvSurf/FindSurface-RealityKit-visionOS-Response-to-Request

The speed of object surface extraction and measurement:

  • Plane: 600 Hz
  • Sphere/cylinder: 300 Hz
  • Cone/torus: 100 Hz.
0
JoonAhn OP
22h

NOTE: With the recent visionOS 2.1 update, we noticed significant changes in the data provided by the MeshAnchor on Apple Vision Pro (The observations are based on our own analysis through our app and are not officially mentioned in Apple's patch notes). Based on our analysis, we recommend the following adjustments to the optimal set of parameters for FindSurface framework for visionOS.

[GitHub Link] https://github.com/CurvSurf/FindSurface-visionOS#optimal-parameter-set-for-apple-vision-pro

0
Geometry recognition and measurement from MeshAnchor
First post date Last post date
 
 
Q