I do have the same problem and filed it under https://feedbackassistant.apple.com/feedback/14485503
Would be great if an engineer could take a look at it, because it is currently causing a lot of issues for my team since we have to pass some accessibility checks, which we can't without Full Keyboard Access working correctly.
Topic:
UI Frameworks
SubTopic:
SwiftUI
Tags: