NavigationStack: Destination not updated on path removal during animation

In my application I am using a custom navigation bar which, besides other functionality, allows the user to navigate back. I am using a NavigationStack with an array with enums as path. On navigating back I simply remove the last element of the path. This is working well as long as the button to navigate back is not clicked while the the navigation stack is not already navigating back i.e. it does not work to quickly click the back button 2+ times. In this case the NavigationStack does not display the destination for the top path, but the previous one. A simple example showing this behavior:

struct ContentView: View {
    @State var navigationPath: [Int] = []

    var body: some View {
        VStack {
            if !navigationPath.isEmpty {
                Button {
                    navigationPath.removeLast()
                } label: {
                    Text("Remove Last")
                }
            }

            NavigationStack(path: $navigationPath) {
                NavigationLink(value: 1) {
                    Text("Navigate to \(1)")
                }
                .navigationDestination(for: Int.self) { value in
                    NavigationLink(value: value+1) {
                        Text("Navigate to \(value+1)")
                    }
                    .toolbar(.hidden, for: .navigationBar)
                }
            }.onChange(of: navigationPath) { newValue in
                print("Path: \(newValue)")
            }
        }
    }
}

I would expect the stack to always show the last element. The path seems to be correct. Does anyone have a solution/workaround for this or does it simply not work to use the navigation path like this?