How to keep bottom toolbar pinned, when sheet's detent are changed in code

I am trying to add a pinned(always visible and not moving) toolbar to the bottom of UISheetPresentationController or presentationDetent.
This is achievable when the user is dragging to change the size of the sheet.

But when changing the detent by code, it does not work. For example, when changing from medium to large, the ViewController's size is changed to large then moved up to the position.
From the user's POV, anything at the bottom to disappear for a few frames then reappears.
From my testing, when using UISheetPresentationController, ViewController only knows the medium and large state's frame and bounds, so manually setting it up won't work.

Is there a way to keep views at the bottom always visible in this case?
The sample code below is in SwiftUI for easy recreation, and the responses can be in UIKit.

Sample Code:

struct ContentView: View {
    
    @State var showSheet: Bool = true
    @State var detent: PresentationDetent = .medium
    
    var body: some View {
        EmptyView()
            .sheet(isPresented: $showSheet) {
                ScrollView {
                    
                }
                .safeAreaInset(edge: .bottom) {
                    Button {
                        detent = (detent == .medium) ? .large : .medium
                    } label: {
                        Text("Change Size")
                    }
                    .buttonStyle(.borderedProminent)
                    .presentationDetents([.medium, .large], selection: $detent)
                    .presentationBackgroundInteraction(.enabled)
                }
            }
    }
}