Window buttons not getting clicked when Scene Colliders Exist

Hi I am using this function to create collisions in my scene from Apple Developer Video I found.

    func processReconstructionUpdates() async {
        for await update in sceneReconstruction.anchorUpdates {
            let meshAnchor = update.anchor
            guard let shape = try? await ShapeResource.generateStaticMesh(from: meshAnchor)
            else {continue}
            
            switch update.event {
            case .added:
                let entity = ModelEntity()
                entity.transform = Transform(matrix: meshAnchor.originFromAnchorTransform)
                entity.collision = CollisionComponent(shapes: [shape], isStatic: true)
                entity.physicsBody = PhysicsBodyComponent()
                entity.components.set(InputTargetComponent())
                
                meshEntities[meshAnchor.id] = entity
                
                contentEntity.addChild(entity)
            case .updated:
                guard let entity = meshEntities[meshAnchor.id] else { fatalError("...") }
                entity.transform = Transform(matrix: meshAnchor.originFromAnchorTransform)
                entity.collision?.shapes = [shape]
            case .removed:
                meshEntities[meshAnchor.id]?.removeFromParent()
                meshEntities.removeValue(forKey: meshAnchor.id)
            }
        }
        
    }

The code works great.

In the same immersive space I am opening a window:

var body: some View {
    RealityView { content in 

         // some other code here

        openWindow(id: "mywindowidhere")
 
         // some other code here
  }
}

The window opens in front of me, but I am not able to click or even hover on the buttons. At first I did not know why that was happening. But then I turned on pointer control and found out that the pointer is actually colliding with the wall. (the window is kinda inside the wall). That is why the pointer never reaches the window and the button never gets clicked.

I initially thought this was a layering issue, but I was not able to find any documentation related to this.

Is this a known issue and is there any way to fix this? Or I am doing anything wrong on my side?