Bounce a virtual object off real world wall?

Hi! While waiting for scene understanding to work in the visionOS simulator, I am trying to bounce a virtual object off a real world wall or other real world object on iOS ;). I load my virtual objects from a Reality Composer file where I set them to participate in physics with dynamic motion type.

With this I am able to have them collide with each other nicely, occlusion also works, but they go right through walls and other real world objects rather than bouncing off...

I've tried a couple of variations of the following code:

    func makeUIView(context: Context) -> ARGameView {
        arView.environment.sceneUnderstanding.options.insert(.occlusion)
        arView.environment.sceneUnderstanding.options.insert(.physics)

        arView.debugOptions.insert(.showSceneUnderstanding)
        
        arView.automaticallyConfigureSession = false
        let config = ARWorldTrackingConfiguration()
        config.planeDetection = [.horizontal, .vertical]
        config.sceneReconstruction = .meshWithClassification
        
        arView.session.run(config)
        
        if let myScene = try? Experience.loadMyScene() {
            ...
            arView.scene.anchors.append(myScene)
        }

        return arView
        
    }

I have found several references that his should "just work", e.g. in https://developer.apple.com/videos/play/tech-talks/609

What am I missing? Testing on iPhone 13 Pro Max with iOS 16.5.1 🤔