SwiftUI List produces random offset when modifying content from a modal view presented above it

Backstory - I wanted to create a SwiftUI List with a collapsible header. Based on some StackOverflow answers for the basic idea, I came up with this (this is just a gist of the component):

GeometryReader { proxy in
    List {
        SwiftUI.Section(
            header: Color.clear
                .frame(height: self.headerSize.height)
                .listRowInsets(EdgeInsets())
                .anchorPreference(key: ScrollOffsetPreferenceKey.self, value: .bounds) {
                    proxy[$0].origin
                }
                .onPreferenceChange(ScrollOffsetPreferenceKey.self, perform: self.handleOffsetChange),
            footer: self.blankFooterView
        ) {}

        self.content
    }
    .environment(\.defaultMinListHeaderHeight, .leastNonzeroMagnitude)
}

In short, there's an empty section the size of the header in the List, based on which I get an offset while scrolling and can move the header accordingly

And it has been working great, except the only problem so far is that if you have a screen with this component and there's another screen presented modally that causes modifications of the content of the underlying screen with a List, this solution produces a random offset value of 39.5333333333333

The value is exactly the same no matter the size of the header I put in there, and I can't see what is causing it. This ONLY happens when modification happens from a modal screen. The style of the List also doesn't matter, bug happens on all of them

Does anyone have an idea of how to fix this? For the full picture, here's a minimal reproducible example: https://github.com/IntegerOverlord/TestCollapsingHeaderBug