VisionOS: Detect plane to place objects issue for animated objects

Hi,

I have used the template code for Plane Detection and placing models on them from here https://developer.apple.com/documentation/visionos/placing-content-on-detected-planes

This source code did not copy the animations in the preview model to the PlacedModel and hence I modified it to do a manual copy of animations and textures. There is a function called materialize() that does this and I was able to modify it to get it working where the placed models are now animating. The issue is when I apply gestures on them like drag or rotate. For those models that go through this logic I'm unable to add gestures even though I'm making sure that Collision and Input Target is set on the Placed Models. Has anyone been able to get this working or is it even a possibility?

My materialize function

func materialize() -> PlacedObject { let shapes = previewEntity.components[CollisionComponent.self]!.shapes

    // Clone render content first as we need its materials
    let clonedRenderContent = renderContent.clone(recursive: true)
    
    print("To be finding main model: \(descriptor.displayName)")
    // Find the main model in preview hierarchy
    func findMainModel(_ entity: Entity) -> Entity? {
        if entity.name == descriptor.displayName.replacingOccurrences(of: " ", with: "_") {
            print("Found main model: \(entity.name)")
            return entity
        }
        for child in entity.children {
            if child.name == descriptor.displayName.replacingOccurrences(of: " ", with: "_") {
                print("Found main model in children: \(child.name)")
                return child
            }
        }
        return nil
    }

    // Clone hierarchy preserving structure, names, and materials
    func cloneHierarchy(_ entity: Entity) -> Entity {
        print("Cloning: \(entity.name)")
        let cloned: Entity
        if let model = entity as? ModelEntity {
            // Clone with recursive false to handle children manually
            cloned = model.clone(recursive: false)

            if let clonedModel = cloned as? ModelEntity,
               let originalMaterials = model.model?.materials {
                // Preserve the original model's materials
                clonedModel.model?.materials = originalMaterials
            }
        } else {
            cloned = Entity()
        }

        // Preserve name and transform
        cloned.name = entity.name
        cloned.transform = entity.transform

        // Clone children
        for child in entity.children {
            let clonedChild = cloneHierarchy(child)
            cloned.addChild(clonedChild)
        }

        return cloned
    }

    print("=== Cloning Preview Structure ===")
    // Clone the preview hierarchy with proper structure
    let clonedStructure = cloneHierarchy(previewEntity)

    // Find and use the main model
    if let mainModel = findMainModel(clonedStructure) {
        print("Using main model for PlacedObject")
        let modelEntity: ModelEntity
        if let asModel = mainModel as? ModelEntity {
            print("Using asModel    ")
            modelEntity = asModel
        } else {
            modelEntity = ModelEntity()
            modelEntity.name = mainModel.name
            // Copy children and transforms
            for child in mainModel.children {
                modelEntity.addChild(child)
            }
            modelEntity.transform = mainModel.transform
        }

        // Add collision component here
        let collisionComponent = CollisionComponent(shapes: shapes, isStatic: false,
                                                  filter: CollisionFilter(group: PlacedObject.collisionGroup, mask: .all))
        modelEntity.components.set(collisionComponent)
        
        // Create the placed object
        let placedObject = PlacedObject(descriptor: descriptor, renderContentToClone: modelEntity, shapes: shapes)
        
        // Set input target on the placed object itself
        placedObject.components.set(InputTargetComponent(allowedInputTypes: [.direct, .indirect]))

        return placedObject
    } else {
        print("Fallback to original render content")
        let placedObject = PlacedObject(descriptor: descriptor, renderContentToClone: clonedRenderContent, shapes: shapes)
        placedObject.components.set(InputTargetComponent(allowedInputTypes: [.direct, .indirect]))
        return placedObject
    }
}

My PlacedObject class where the init has the recursive cloning removed because it is handled in materialize

class PlacedObject: Entity { let fileName: String

// The 3D model displayed for this object.
private let renderContent: ModelEntity

static let collisionGroup = CollisionGroup(rawValue: 1 << 29)

// The origin of the UI attached to this object.
// The UI is gravity aligned and oriented towards the user.
let uiOrigin = Entity()

var affectedByPhysics = false {
    didSet {
        guard affectedByPhysics != oldValue else { return }
        if affectedByPhysics {
            components[PhysicsBodyComponent.self]!.mode = .static
        } else {
            components[PhysicsBodyComponent.self]!.mode = .static
        }
    }
}


var isBeingDragged = false {
    didSet {
        affectedByPhysics = !isBeingDragged
    }
}

var positionAtLastReanchoringCheck: SIMD3<Float>?

var atRest = false

init(descriptor: ModelDescriptor, renderContentToClone: ModelEntity, shapes: [ShapeResource]) {
    fileName = descriptor.fileName

// renderContent = renderContentToClone.clone(recursive: true) renderContent = renderContentToClone super.init() name = renderContent.name

    // Apply the rendered content’s scale to this parent entity to ensure
    // that the scale of the collision shape and physics body are correct.
    scale = renderContent.scale
    renderContent.scale = .one
    
    // Make the object respond to gravity.
    let physicsMaterial = PhysicsMaterialResource.generate(restitution: 0.0)
    let physicsBodyComponent = PhysicsBodyComponent(shapes: shapes, mass: 1.0, material: physicsMaterial, mode: .static)
    components.set(physicsBodyComponent)
    components.set(CollisionComponent(shapes: shapes, isStatic: false,
                                      filter: CollisionFilter(group: PlacedObject.collisionGroup, mask: .all)))
    

    addChild(renderContent)
    addChild(uiOrigin)
    uiOrigin.position.y = extents.y / 2 // Position the UI origin in the object’s center.
    
    // Allow direct and indirect manipulation of placed objects.
    components.set(InputTargetComponent(allowedInputTypes: [.direct, .indirect]))
    
    
    // Add a grounding shadow to placed objects.
    renderContent.components.set(GroundingShadowComponent(castsShadow: true))
    
}


required init() {
    fatalError("`init` is unimplemented.")
}

}

Thanks

Hi @RaviVizspatial

I ran the sample project with your changes and I'm able to drag the placed objects. You mentioned rotate, but the sample you referenced doesn't include rotation and neither do your changes. Can you elaborate on the issue? Are there more changes? If so, please push a project that recreates the issue somewhere I can pull it.

As an aside, I'd consider refactoring the code to load copies of the entity to avoid the complexity of cloning the entity hierarchy and its components. For example, instead of loading a USD file into an entity and then cloning that entity, load the USD file again to create a new entity.

I'll keep an eye on this thread. I encourage you to followup.

Hi,

Thanks for trying this. Did your 3D model have animation in it? Are you saying that when you place the model, it plays the animation and then it also allows you to drag that animating model and then place it again? Thanks

P.S. This method of cloning the model to show a preview and then place it from the Vision OS's sample code from the developer site.

Hi @RaviVizspatial

Good call out. I did not use an animating 3D model, but I should have. Is the animation baked into a USD file? I just tested an animating USD and its entity's collision shape did not follow the animation. Can you visualize your collision shapes and tell me if they look as expected? Here's how to use Xcode to do that:

  • In debug mode open "Debug Visualizations" by clicking the button to the right of the "step out" button in the "Debug Area" (bottom pane). It kind of looks like a box.
  • Select "Collision Shapes and Axes".
  • Doing this will outline the collision shapes around the entities in your application.
  • Verify your collision shapes are correct.

Please let me know how it goes.

Hi

Yes, the animation is baked into the USD file. I followed your advice and it does have an issue with collision. The model in preview i.e. before placement shows collision with white lines but the one after does not have the white lines. I do have the section where collision component is created but is there something I'm doing wrong? Thanks

VisionOS: Detect plane to place objects issue for animated objects
 
 
Q