Animating Removal From LazyVGrid inside ScrollView Issue

I have a two column LazyVGrid inside a ScrollView and want to animate the removal of an item when the user taps on it. This seems to work fine when either the top or the bottom of the grid is visible (i.e. scrolled all the way up or down) and when removing an item doesn't change the number of rows. But when you scroll down a bit and remove an item when it results in the number of rows changing, it flashes as it's rendering.

Minimally Reproducible Example

import SwiftUI

extension Color {
    static var random: Color {
        let red = Double.random(in: 0...1)
        let green = Double.random(in: 0...1)
        let blue = Double.random(in: 0...1)
        return Color(red: red, green: green, blue: blue)
    }
}

struct ContentView: View {
    @State private var items: [Int] = Array(1...50)
    
    let columns = [GridItem(.flexible(), spacing: 2), GridItem(.flexible(), spacing: 2)]
    var colorDictionary: [Int: Color] = [:]

    init() {
        for i in 1...50 {
            colorDictionary[i] = Color.random
        }
    }
    
    var body: some View {
        ScrollView {
            LazyVGrid(columns: columns, spacing: 2) {
                ForEach(items, id:\.self) { item in
                    ZStack {
                        colorDictionary[item]
                        Text("\(item)")
                    }
                    .frame(width: 180, height: 90)
                    .overlay(Rectangle().stroke())
                    .onTapGesture {
                        withAnimation {
                            items.removeAll { $0 == item }
                        }
                    }
                }
            }
        }
    }
}

On iPhone 15 Pro simulator, if you scroll down until 22 is approximately in the middle, and then tap on that location a number of times in a row and you should see the flash every second tap.

What am I doing wrong? How can I always smoothly animate the removal of a view from a LazyVGrid inside a ScrollView? Without the ScrollView, it seems to animate fine always despite not being able to see the top and bottom of the grid.