Billboard Entity with AttachmentView

Question

JustinML OP

Created 2d

Replies 4

Boosts 0

Participants 2

Hey Everyone, Happy New Year!

I wanted to see if you have seen this before. I have added an attachment to the RealityView as a child on an entity that has a Billboard component set on it. I wanted to create the effect that the attachment is offset by .5 meters from center and follows the device as you move around it. IT works great until you try click a button.

The attachment moves with the billboard, but the collision box around the attachment is not following it. If I position myself perfectly it works.

Screenshot 2025-01-03 at 2.29.02 PM.png

Video Example: https://youtu.be/4d9Vx7K8MmU

//
//  ImmersiveView.swift
//  Billboard Attachment
//
//  Created by Justin Leger on 1/3/25.
//

import SwiftUI
import RealityKit
import RealityKitContent

struct ImmersiveView: View {
    
    var rootEntity = Entity()

    var body: some View {
        RealityView { content, attachments in
            // Add the initial RealityKit content
            
            let sphereEntity = ModelEntity(mesh: .generateSphere(radius: 0.1), materials: [SimpleMaterial(color: .red, roughness: 1, isMetallic: false)])
            sphereEntity.position = [0.0, 1.0, -2.0]
            
            let controlsPivotEntity = Entity()
            controlsPivotEntity.components[BillboardComponent.self] = .init()
            
            // Extract the attachemnt entity and disable it before its used.
            if let controlsViewAttachmentEntity = attachments.entity(for: PlacedThingControls.attachmentId) {
                controlsViewAttachmentEntity.position.z = 0.5
                controlsPivotEntity.addChild(controlsViewAttachmentEntity)
                sphereEntity.addChild(controlsPivotEntity)
            }
            
            content.add(sphereEntity)
        }
        attachments: {
            Attachment(id: PlacedThingControls.attachmentId) {
                PlacedThingControls()
            }
        }
    }
}

#Preview(immersionStyle: .mixed) {
    ImmersiveView()
        .environment(AppModel())
}

struct PlacedThingControls: View {
    static let attachmentId = "placed-thing-3D-controls"
    
    var body: some View {
        VStack {
            HStack(spacing: 0) {
                Button {
                    print("🗺️🗺️🗺️ Map selected pieces")
                } label: {
                    Text("\(Image(systemName: "plus.square.dashed")) Manage Mesh Maps")
                        .fontWeight(.semibold)
                        .frame(maxWidth: .infinity)
                }
                .padding(.leading, 20)
                
                Spacer()
                
                Button(role: .destructive) {
                    print("🗑️🗑️🗑️ Delete selected pieces")
                } label: {
                    Label {
                        Text("Delete")
                    } icon: {
                        Image(systemName: "trash")
                    }
                    .labelStyle(.iconOnly)
                }
                .padding(.trailing, 20)
            }
            .padding(.vertical)
            .frame(minWidth: 320, maxWidth: 480)
        }
        .glassBackgroundEffect()
    }
}

Boost

Answer 1

Vision Pro Engineer OP

Apple

2d

Hi @JustinML

Thanks for the detailed post, it really helps. This behavior seems unexpected. I'd appreciate if you file a bug report via feedback assistant and reply with the FB number. Bug Reporting: How and Why? has tips on creating a bug report.

Depending on your use case, you may consider using a volume with a toolbar. This creates controls that follow a person as they move around a volume. Dive deep into volumes and immersive spaces demonstrates this.

0

Answer 2

JustinML OP

2d

I will definitly file a bug report on this. Thank you very much.

I am aware of that feature of a volume, but unfortunately this is a simplified excert from a much larger mixed reality project that we are trying to control to selected entities. This code is only to illustrate and reproduce the issue we are seeing.

Can you think of a way I can work around this in the meantime? is there another way I can achieve a offset pivot point?Maybe without using the billboard component?

0

Answer 3

JustinML OP

2d

Feedback filed - FB16233297

0

Answer 4

JustinML OP

1d

Found a workaround by using the Billboard ECS from the VisionOS 1.0 version SwiftSplash. Came across the source on stack overflow. I made a few naming changes, but it fixes the issue.

Is there a better solution to this?

https://stackoverflow.com/questions/60577468/how-to-implement-a-billboard-effect-lookat-camera-in-realitykit

//
//  BillboardAttachmentFixSystem.swift
//
//  Created by Justin Leger on 1/3/25.
//


import Foundation
import ARKit
import RealityKit
import SwiftUI
import simd
import OSLog

/// An ECS system that points all entities containing a billboard component at the camera.
public struct BillboardAttachmentFixSystem: System {
    
    static let query = EntityQuery(where: .has(BillboardAttachmentFixComponent.self))
    
    private let arkitSession = ARKitSession()
    private let worldTrackingProvider = WorldTrackingProvider()
    
    public init(scene: RealityKit.Scene) {
        setupARKitSession()
    }
    
    func setupARKitSession() {
        Task {
            do {
                try await arkitSession.run([worldTrackingProvider])
            } catch {
                os_log(.info, "Error: \(error)")
            }
        }
    }
    
    public func update(context: SceneUpdateContext) {
        
        let entities = context.scene.performQuery(Self.query).map({ $0 })
        
        guard !entities.isEmpty,
              let pose = worldTrackingProvider.queryDeviceAnchor(atTimestamp: CACurrentMediaTime()) else { return }
        
        let cameraTransform = Transform(matrix: pose.originFromAnchorTransform)
        
        for entity in entities {
            entity.look(at: cameraTransform.translation,
                        from: entity.scenePosition,
                        relativeTo: nil,
                        forward: .positiveZ)
        }
    }
}

/// The component that marks an entity as a billboard object which will always face the camera.
public struct BillboardAttachmentFixComponent: Component, Codable {
    public init() {}
}


/// Entity extension holding convenience accessors and mutators for the components
/// this system uses. Components are stored in the `components` dictionary using the
/// component class (`.self`) as the key. This adds calculated properties to allow setting
/// and getting these components.
extension Entity {
    /// Property for getting or setting an entity's `MockThruBillboardComponent`.
    var mockThruBillboardComponent: BillboardAttachmentFixComponent? {
        get { components[BillboardAttachmentFixComponent.self] }
        set { components[BillboardAttachmentFixComponent.self] = newValue }
    }
}

0