ARKit Planes do not appear where expected on visionOS

I think figured out what I was doing wrong. I was using the extents of the anchor to create meshes, then placing the meshes at the transform for the anchor. I was expecting an anchor plane to be something I could turn into a geometric plane.

Diving deeper into it, anchors are not planes the sense that a plane mesh is. These anchors are actually n-gons that don't necessarily line up with shape of a plane. Apple has an example project that creates these, but applies an occlusion material. I swapped that for a material with random colors so I could visualize what is happening. Each anchor has a n-gon, represented with meshVertices. The example project used some extensions to use that data to create shapes for the meshes.

Personally, I found the example project difficult to understand. It has way too much abstraction and stashes the good stuff in extensions.

Here is a modified version of the example from the first post, without the abstractions from the Apple example project.

I'd love to hear any more ideas from you folks. How can we improve this?

import SwiftUI
import RealityKit
import RealityKitContent
import ARKit

struct Example068: View {
    @State var session = ARKitSession()
    @State private var planeAnchors: [UUID: Entity] = [:]
    @State private var planeColors: [UUID: Color] = [:]

    var body: some View {
        RealityView { content in

        } update: { content in
            for (_, entity) in planeAnchors {
                if !content.entities.contains(entity) {
                    content.add(entity)
                }
            }
        }
        .task {
            try! await setupAndRunPlaneDetection()
        }
    }

    func setupAndRunPlaneDetection() async throws {
        let planeData = PlaneDetectionProvider(alignments: [.horizontal, .vertical, .slanted])
        if PlaneDetectionProvider.isSupported {
            do {
                try await session.run([planeData])
                for await update in planeData.anchorUpdates {
                    switch update.event {
                    case .added, .updated:
                        let anchor = update.anchor
                        if planeColors[anchor.id] == nil {
                            planeColors[anchor.id] = generatePastelColor()
                        }
                        let planeEntity = createPlaneEntity(for: anchor, color: planeColors[anchor.id]!)
                        planeAnchors[anchor.id] = planeEntity
                    case .removed:
                        let anchor = update.anchor
                        planeAnchors.removeValue(forKey: anchor.id)
                        planeColors.removeValue(forKey: anchor.id)
                    }
                }
            } catch {
                print("ARKit session error \(error)")
            }
        }
    }
    
    private func generatePastelColor() -> Color {
        let hue = Double.random(in: 0...1)
        let saturation = Double.random(in: 0.2...0.4)
        let brightness = Double.random(in: 0.8...1.0)
        return Color(hue: hue, saturation: saturation, brightness: brightness)
    }
    
    private func createPlaneEntity(for anchor: PlaneAnchor, color: Color) -> Entity {
        let entity = Entity()
        entity.name = "Plane \(anchor.id)"
        entity.setTransformMatrix(anchor.originFromAnchorTransform, relativeTo: nil)

        // Generate a mesh for the plane (for occlusion).
        var meshResource: MeshResource? = nil
        do {
            var contents = MeshResource.Contents()
            contents.instances = [MeshResource.Instance(id: "main", model: "model")]
            var part = MeshResource.Part(id: "part", materialIndex: 0)
            
            // Convert vertices to SIMD3<Float>
            let vertices = anchor.geometry.meshVertices
            var vertexArray: [SIMD3<Float>] = []
            for i in 0..<vertices.count {
                let vertex = vertices.buffer.contents().advanced(by: vertices.offset + vertices.stride * i).assumingMemoryBound(to: (Float, Float, Float).self).pointee
                vertexArray.append(SIMD3<Float>(vertex.0, vertex.1, vertex.2))
            }
            part.positions = MeshBuffers.Positions(vertexArray)
            
            // Convert faces to UInt32
            let faces = anchor.geometry.meshFaces
            var faceArray: [UInt32] = []
            let totalFaces = faces.count * faces.primitive.indexCount
            for i in 0..<totalFaces {
                let face = faces.buffer.contents().advanced(by: i * MemoryLayout<Int32>.size).assumingMemoryBound(to: Int32.self).pointee
                faceArray.append(UInt32(face))
            }
            part.triangleIndices = MeshBuffer(faceArray)
            
            contents.models = [MeshResource.Model(id: "model", parts: [part])]
            meshResource = try MeshResource.generate(from: contents)
        } catch {
            print("Failed to create a mesh resource for a plane anchor: \(error).")
        }

        var material = PhysicallyBasedMaterial()
        material.baseColor.tint = UIColor(color)

        if let meshResource {
            entity.components.set(ModelComponent(mesh: meshResource, materials: [material]))
        }

        return entity
    }
}

#Preview {
    Example068()
}

You can use simple planes if you want but you have to account for the fact that the anchor is not necessarily the center of the plane.

You will still get a "bounding box" style representation and there may be some over/under lap but it should be much closer to what you were expecting.

Something like the following should work for you on both vertical and horizontal planes.

The relevant line is the setTransformationMatrix

entity.setTransformMatrix(
  anchor.originFromAnchorTransform * anchor.geometry.extent.anchorFromExtentTransform,
  relativeTo: nil)

if generateSimplifiedPlanes {
  let mesh: MeshResource = try! await MeshResource.generatePlane(
  width: anchor.geometry.extent.width, 
  height: anchor.geometry.extent.height)

modelEntity.model?.mesh = mesh
                            
modelEntity.collision?.shapes = [try! await ShapeResource.generateStaticMesh(from: mesh)]
entity.setTransformMatrix(
  anchor.originFromAnchorTransform * anchor.geometry.extent.anchorFromExtentTransform, 
  relativeTo: nil)
}
else {
  let shape: ShapeResource = try! await ShapeResource.generateStaticMesh(
    positions: anchor.geometry.meshVertices.asSIMD3(ofType: Float.self),
    faceIndices: anchor.geometry.meshFaces.asUInt16Array())
                            
  modelEntity.collision?.shapes = [shape]
  modelEntity.model?.mesh = planeAnchorToMeshResource(anchor)
  entity.transform = Transform(matrix: anchor.originFromAnchorTransform)
}

Notes: It appears that the extent is always defined on a vertical plane so if you apply anchorFromExtentTransform you always want to use a vertical plane.

This is rough code from a while ago and I haven't checked to see if there are newer/easier ways to do this.

Hi @radicalappdev

@PlatformGoblin thanks for helping out. I agree, anchorFromExtentTransform is the missing piece of the puzzle. The code to convert a PlaneAnchor to a ModelEntity with collision shapes can be simplified to:

let extent = anchor.geometry.extent
let mesh = MeshResource.generatePlane(width: extent.width, height: extent.height)
let material = SimpleMaterial(color: .green, isMetallic: false)
let entity = ModelEntity(mesh: mesh, materials: [material])
entity.generateCollisionShapes(recursive: false)
entity.transform = Transform(matrix: matrix_multiply(anchor.originFromAnchorTransform, extent.anchorFromExtentTransform))