Simultaneously interacting with SwiftUI View below a UIViewRepresentable

My problem

I have a SwiftUI app that uses PKCanvasView to allow drawing. The PKCanvasView is transparent and there are SwiftUI views (e.g. an Image) behind which can be dragged and opened.

I want the user to be able to draw on the PKCanvasView with their apple pencil and interact with the views below with their finger...

I can't figure out how to enable simultaneous interaction with the PKCanvasView (i.e. if using apple pencil draw) and the swiftUI views behind (i.e. if using finger, drag and move views).

The context

Here is my PKCanvasView wrapped in a UIViewRepresentable struct:

Code Block 
struct CanvasView: UIViewRepresentable {
    
    @Binding var canvasView: PKCanvasView
    
    @State var toolPicker = PKToolPicker()
    
    func makeUIView(context: Context) -> PKCanvasView {
        
        canvasView.drawingPolicy = .pencilOnly
        canvasView.backgroundColor = .clear
        canvasView.isOpaque = false
        canvasView.alwaysBounceVertical = true
        canvasView.alwaysBounceHorizontal = true
        
        
        toolPicker.setVisible(true, forFirstResponder: canvasView)
        toolPicker.addObserver(canvasView)
        
        return canvasView
    }
    
    func updateUIView(_ uiView: PKCanvasView, context: Context) {
    }
    
}

The structure of my content view is as follows:

Code Block swift
ZStack(alignment: .topLeading){
                CanvasView(canvasView: $canvasView)
                    .zIndex(0)
                ForEach(canvas.subCanvases.indices, id: \.self){ index in
                        CanvasCardView(
                            canvas: $canvas.subCanvases[index],
                            animationNamespace: animationNamespace,
                            onReorder: {
                                reorderCard(at: canvas.subCanvases[index].zOrder)
                        })
                }
            }

The CanvasCardView are the background SwiftUI views which can be dragged and opened. The view attaches a negative zIndex to them which orders them behind the CanvasView (PKCanvasView).

My attempts

I've tried two approaches so far which don't work:

1. Subclassing PKCanvasView to override point(inside:with:). Passing false when I want to touches to be passed through. Problem with this is that the swiftUI views never recognise the touches through the UIView. I assume as there are no UITouchs behind the scenes.

2. Adding a @State property to my contentView that can update an allowsHitTesting() modifier dynamically when required. The issue with this is that without allowing the PKCanvasView to be "hit" I can't determine whether it is a pencil touch or not. So i can't properly interact with PKCanvasView simultaneously.

Any other ideas if this is possible? That is, passing touches through a UIViewRepresentable dynamically?

I really want to avoid shoving the SwiftUI views into a UIView (with UIHostingController) and then turning them back into SwiftUI.

Many thanks!