Non-convex collision?

Hi!

I think this should be a pretty normal usage of ARKit / RealityKit

I have a static mesh for my environment, that I want to have static collision properties.

My options for making this interact with dynamic bodies are:

  1. ShapeResource.generateConvex(...) -- which overshoots my shape dramatically.
  2. Entity.generateCollisionShapes(...) which also overshoots.

I notice additional APIs around ShapeResource -- ShapeResource.generateStaticMesh(positions:faceIndices) seems to be exactly what I need.

So far, I haven't been able to invoke this successfully to set my collision box.

Questions:

  • Is this not, a completely normal thing for developers to want to do? Why is there no support for this out-of-the-box in RealityKit/ARKit?
  • To support this in my app, everywhere I've read has said I need to parse the .obj of my terrain manually, and find triangulated faces and pipe them into this function. That feels like a very standardized process -- and given that RealityKit is already forcing me to use .usdz, why should this not be a part of the SDK?
  • Regardless- I triangulated my terrain mesh, and have been working on parsing code to get the positions and faceIndices for this set up (as an extension on Entity).

Is this the right approach? Am I missing something more obvious?

Thanks, Justin

@JayDev85 Are you able to access the mesh vertices, indices, etc? Worst case you could create convex meshes from the concave mesh yourself.

Are you able to access the mesh vertices, indices, etc with the API? Worst case you could create convex meshes from the concave mesh yourself.

so...

If you have a ModelEntity, you can get the .model which contains a mesh (https://developer.apple.com/documentation/realitykit/modelcomponent/mesh)

The mesh contains this contents struct (https://developer.apple.com/documentation/realitykit/meshresource/contents-swift.struct) , and eventually all the way down to the MeshResource.Part parts (https://developer.apple.com/documentation/realitykit/meshresource/part). Then you start to have index / triangle data.

By this point I feel pretty in-over-my-head....

What has worked is simply going into my scene in RealityComposer and creating a bunch of cubes with collision / physics, and placing them around the model / setting a transparent material. If you're willing to spend 3 hours lining your scene with cubes manually, this does work for simple concave-ish geometries.

There has to be a better solution...

The solution, I think, is to decompose the concave mesh into convex meshes. Then, if this is a static mesh, then you're in luck because optimal performance doesn't matter much if you just want a result in a reasonable amount of time. Resave it as a collection of meshes for reloading in the future. If it's a dynamic mesh, you're kind of stuck doing this at runtime. I think this is a very normal thing to do. Concave collision detection is more expensive than convex.

Note: I think it would be useful to include some built-in algorithm for handling this both in Metal Performance and RealityKit APIs. (Maybe file a feature request?)

In general, decomposing a concave object into a set of convex objects is difficult. I.e. it's NP-Hard. There is lots of literature about this, including some approximation algorithms. Search the web.

But I note that the original question referred to "terrain". Is this concave object terrain? That is an interesting special case which may be easier to solve.

Note that I know next-to-nothing about ARKit / RealityKit, but I do know a fair bit about geometry algorithms in general.

The standard way to use a concave collision shape in RealityKit would be to use ShapeResource.generateStaticMesh. New in 2024, it's easier than ever to use generateStaticMesh by converting a MeshResource to a ShapeResource. Keep in mind that this is a computationally intensive operation, and it has some limitations. In particular, an Entity can not move in response to physics forces (e.g. gravity) when it contains a generateStaticMesh-based shape.

Another approach is to use convex proxies (via ShapeResource.generateConvex, generateBox, and so on). As helpfully mentioned by @endecotp, the decomposition of a concave 3D shape into convex collision proxies is a nontrivial and computationally intensive problem. Ideally, you would perform an approximate convex decomposition ahead of time to save power and improve performance. There are open source solutions which can help you with this such as v-HACD.

Finally, if there are specific features or APIs you would like to see in RealityKit in the future, it would be helpful to us if you provided details in Feedback Assistant.

Non-convex collision?
 
 
Q