Found out the cause of the crashes. We sync the Package.resolved file of the Xcode project with some local swift packages contained in the project by copying it to the root of the packages, next to Package.swift. This means that the Package.resolved would have more entries than it should for the smaller packages.
This worked fine in Xcode 15, but it looks like Xcode 16 expects the dependency count to be exact.