Focusable doesn't work on iPad with external keyboard

I have a custom input view in my app which is .focusable(). It behaves similar to a TextField, where it must be focused in order to be used.

This works fine on all platforms including iPad, except when when an external keyboard is connected (magic keyboard), in which case it can't be focused anymore and becomes unusable.

Is there a solution to this, or a workaround? My view is very complex, so simple solutions like replacing it with a native view isn't possible, and I must be able to pragmatically force it to focus.

Here's a very basic example replicating my issue. Non of the functionality works when a keyboard is connected:

struct FocusableTestView: View {
    @FocusState private var isRectFocused: Bool

    var body: some View {
        VStack {
            // This text field should focus the custom input when pressing return:
            TextField("Enter text", text: .constant(""))
                .textFieldStyle(.roundedBorder)
                .onSubmit {
                    isRectFocused = true
                }
                .onKeyPress(.return) {
                    isRectFocused = true
                    return .handled
                }

            // This custom "input" should focus itself when tapped:
            Rectangle()
                .fill(isRectFocused ? Color.accentColor : Color.gray.opacity(0.3))
                .frame(width: 100, height: 100)
                .overlay(
                    Text(isRectFocused ? "Focused" : "Tap me")
                )
                .focusable(true, interactions: .edit)
                .focused($isRectFocused)
                .onTapGesture {
                    isRectFocused = true
                    print("Focused rectangle")
                }
            
            // The focus should be able to be controlled externally:
            Button("Toggle Focus") {
                isRectFocused.toggle()
            }
            .buttonStyle(.bordered)
        }
        .frame(maxWidth: .infinity, maxHeight: .infinity, alignment: .center)
    }
}