Using ARView's project(_:) method to convert to screen coordinates.

I'm trying to understand how to use the project(_:) function provided by ARView to convert 3D model coordinates to 2D screen coordinates, but am getting unexpected results.

Below is the default Augmented Reality App project, modified to have a single button that when tapped will place a circle over the center of the provided cube. However, when the button is pressed, the circle's position does not line up with the cube.

I've looked at the documentation for project(_:), but it doesn't give any details about how to convert a point from model coordinates to "the 3D world coordinate system of the scene". Is there better documentation somewhere on how to do this conversion?

//  ContentView.swift
import SwiftUI
import RealityKit

class Coordinator {
    var arView: ARView?
    var anchor: AnchorEntity?
    var model: Entity?
}

struct ContentView : View {
    @State var coord = Coordinator()
    @State var circlePos = CGPoint(x: -100, y: -100)

    var body: some View {
        ZStack {
            ARViewContainer(coord: coord).edgesIgnoringSafeArea(.all)

            VStack {
                Spacer()
                 
                Circle()
                    .frame(width: 10, height: 10)
                    .foregroundColor(.red)
                    .position(circlePos)

                Button(action: { showMarker() }, label: { Text("Place Marker") })
            }
        }
    }
    
    func showMarker() {
        guard let arView = coord.arView else { return }
        guard let model = coord.model else { return }
        guard let anchor = coord.anchor else { return }
        
        print("Model position is: \(model.position)")
        
        // convert position into anchor's space
        let modelPos = model.convert(position: model.position, to: anchor)
        print("Converted position is: \(modelPos)")

        // convert model locations to screen coordinates
        circlePos = arView.project(modelPos) ?? CGPoint(x: -1, y: -1)
        print("circle position is now \(circlePos)")
    }
}

struct ARViewContainer: UIViewRepresentable {
    var coord: Coordinator
    
    func makeUIView(context: Context) -> ARView {
        
        let arView = ARView(frame: .zero)
        coord.arView = arView

        // Create a cube model
        let mesh = MeshResource.generateBox(size: 0.1, cornerRadius: 0.005)
        let material = SimpleMaterial(color: .gray, roughness: 0.15, isMetallic: true)
        let model = ModelEntity(mesh: mesh, materials: [material])
        coord.model = model

        // Create horizontal plane anchor for the content
        let anchor = AnchorEntity(.plane(.horizontal, classification: .any, minimumBounds: SIMD2<Float>(0.2, 0.2)))
        anchor.children.append(model)
        coord.anchor = anchor

        // Add the horizontal plane anchor to the scene
        arView.scene.anchors.append(anchor)

        return arView
        
    }
    
    func updateUIView(_ uiView: ARView, context: Context) {}
    
}

#Preview {
    ContentView(coord: Coordinator())
}
Answered by fraggle in 770252022

To get a point in 3D world coordinates for use with project(_:), use Entity's convert(position:from:) instance method. The description of convert(position:from:)'s referenceEntity says "Set this to nil to indicate world space.".

In the example above, the code:

        // convert position into anchor's space
        let modelPos = model.convert(position: model.position, to: anchor)

Should be changed to:

        // convert position into world coordinate space
        let modelPos = model.convert(position: model.position, to: nil)
Accepted Answer

To get a point in 3D world coordinates for use with project(_:), use Entity's convert(position:from:) instance method. The description of convert(position:from:)'s referenceEntity says "Set this to nil to indicate world space.".

In the example above, the code:

        // convert position into anchor's space
        let modelPos = model.convert(position: model.position, to: anchor)

Should be changed to:

        // convert position into world coordinate space
        let modelPos = model.convert(position: model.position, to: nil)
Using ARView's project(_:) method to convert to screen coordinates.
 
 
Q