RoomPlan/UIKit/ARKit - add custom text to UI

Hi, if you want to display a 2D UI, then using UIKit is the best approach (e.g. show/update text in a UILabel). If you want to display a 3D object, you would need to use ARKit. You can access the arSession from your RoomCaptureSession and add an ARAnchor to attach your content to.

Hi Frameworks Engineer Team,

Thanks for your reply.

Following your answer, I'm trying to display my content using the arSession from my RoomCaptureSession.

I developed this methods to add custom content on detected objects :

func captureSession(_ session: RoomCaptureSession, didAdd room: CapturedRoom) {
     
    for object in room.objects {
      print("Trying to add anchorText for category : ", object.category)
      self.drawBox(session: session, dimensions: object.dimensions,
             transform: object.transform, confidence: object.confidence)
    }
}

func drawBox(session: RoomCaptureSession, dimensions: simd_float3, transform: float4x4, confidence: CapturedRoom.Confidence) {
     
    print("update scan log drawBox() method")

    let box = MeshResource.generateBox(width: dimensions.x,
                      height: dimensions.y,
                     depth: dimensions.z > 0 ? dimensions.z : 0.1)
     
    let boxEntity = ModelEntity(mesh: box)
     
    let textEntity = ModelEntity(mesh: .generateText("My custom text",
               extrusionDepth: 0.01,
                font: .init(name: "Helvetica", size: 0.5)!,
               containerFrame: .zero,
                  alignment: .center,
                lineBreakMode: .byWordWrapping))
       
    // Changing a position and orientation of the parent
    boxEntity.addChild(textEntity)
    boxEntity.position.x = dimensions.x
    boxEntity.position.y = dimensions.y
    boxEntity.position.z = dimensions.z
     
    let anchor = AnchorEntity()
    anchor.transform = Transform(matrix: transform)

    boxEntity.setParent(anchor)
     
    let arAnchor = ARAnchor(transform: anchor.transform.matrix)
     
    session.arSession.add(anchor: arAnchor)

but my application is crashing on the line 'let box = MeshResource.generateBox' with the following error : 'Thread 25: EXC_BREAKPOINT (code=1, subcode=0x1034cfd74)'

I google it but didn't find anything...

• Could you give me some explanations about this error ?
• Does the rest of the code is good ? The end looks strange with transforms :/

Thanks in advance for your reply and happy new year :)

Goat.

Hello,

The ARAnchor you're adding in the end of your code snippet has no effect; it would make more sense to add that first, and then create an AnchorEntity(anchor:) using the existing ARAnchor (see https://developer.apple.com/documentation/realitykit/anchorentity/init(anchor:)). That way, your content will be tied to the ARAnchor and benefit from pose graph updates.

With regard to the MeshResource error, please check on which thread this code is executed. RealityKit mesh generation APIs need to run on the main thread, so you may need to dispatch it to DispatchQueue.main.

Hello,

Thanks for your reply, DispatchQueue made the trick for MeshResource problem.

For ARAnchor subject, can you confirm that I'm doing the right steps :

Step 1. Creating ARAnchor using CapturedRoom.Objects[] and add it to ARSession :

for object in room.objects {
    let anchor = RoomObjectAnchor(object) // RoomObjectAnchor is a custom Object extending ARAnchor class
    session.arSession.add(anchor: anchor)
}

1. Creating my custom AnchorEntity attached to ARAnchor created :

let anchorEntity = AnchorEntity(anchor: anchor)
let textEntity = ModelEntity(mesh: .generateText("My custom text",
               extrusionDepth: 0.01,
                font: .init(name: "Helvetica", size: 0.5)!,
               containerFrame: .zero,
                 alignment: .center,
                lineBreakMode: .byWordWrapping))
anchorEntity.addChild(textEntity)

Does I have to make a third step to add the AnchorEntity to the arSession ?

From my research and test, I didn't find any ways to add my AnchorEntity to the arSession directly, so what is the best way to add our AnchorEntity ?

The only way I found is this one but I'd prefer to use the arSession directly :

arView.scene.addAnchor(anchorEntity)

Thanks in advance for your reply,

Goat.

What you are doing looks correct. There is no need (and also no way) to add an entity to an ARSession.

Entities are elements of a RealityKit scene, so they are part of the rendering system. ARAnchors are objects that represent the position of a virtual object in an ARKit session - they are unaware of the rendering layer used on top (instead of RealityKit, that could also be SceneKit, Metal, etc). By creating an AnchorEntity(anchor: anchor) you establish the link between by the RealityKit entity and the ARAnchor. Whenever ARKit updates the position of the anchor, the associated AnchorEntity will be updated as well.

Thanks a lot, all work fine now, I can add my custom text on detected obejcts from RoomPlan API.

Last question : To add my custom text to the arView, I had to change my main view from UIView to ARView and so I lost all instructions users from RoomCaptureView. Is there any way to implements those instructions to my ARView ?

To have more details, my main class looks like this :

class ViewController: UIViewController, RoomCaptureSessionDelegate {

 @IBOutlet var arView: ARView!
var captureSession: RoomCaptureSession?

private func startSession() {
  captureSession?.run(configuration: roomCaptureSessionConfig)
 }

func captureSession(
  _ session: RoomCaptureSession, didStartWith configuration: RoomCaptureSession.Configuration
 ) {
    
  arView.session.pause()
  arView.session = session.arSession
  arView.session.delegate = self

 }

otherMethods()
.
.
.

}

Thanks in advance,

Goat.

Hi Keeedl In your side export data working fine. Please share that. I'm facing below issue please review https://developer.apple.com/forums/thread/726918