xcodebuild failing when package plugin is added to project

I have created a build tool plugin in one of my SPM packages, and am trying to get it working in my project. It works fine when I build from Xcode, or have at least built the project in Xcode once before with the plugin.

But if I try to build the project using xcodebuild on a machine where I have never built the project before, it fails with this error:

error: '2.3.0': Invalid manifest (compiled with: ["/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/swiftc", "-vfsoverlay", "/var/folders/5_/q4yl04gs2kld1zztqxkqjdgh0000gq/T/TemporaryDirectory.BWwJWG/vfs.yaml", "-L", "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/pm/ManifestAPI", "-lPackageDescription", "-Xlinker", "-rpath", "-Xlinker", "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/pm/ManifestAPI", "-target", "arm64-apple-macosx14.0", "-sdk", "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX15.5.sdk", "-F", "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/Library/Frameworks", "-F", "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/Library/PrivateFrameworks", "-I", "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/usr/lib", "-L", "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/usr/lib", "-swift-version", "5", "-I", "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/pm/ManifestAPI", "-sdk", "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX15.5.sdk", "-package-description-version", "5.7.0", "/private/var/folders/5_/q4yl04gs2kld1zztqxkqjdgh0000gq/T/SwiftTemplate/1FA05B4D-851D-4D2B-ADD6-E5A0DF70CD37/2.3.0/Package.swift", "-o", "/var/folders/5_/q4yl04gs2kld1zztqxkqjdgh0000gq/T/TemporaryDirectory.HUxmAq/2.3.0-manifest"])
<unknown>:0: error: error opening '/var/folders/5_/q4yl04gs2kld1zztqxkqjdgh0000gq/C/clang/ModuleCache/PackageDescription-3TZGMDBKTLI5E.swiftmodule' for output: /var/folders/5_/q4yl04gs2kld1zztqxkqjdgh0000gq/C/clang/ModuleCache/PackageDescription-3TZGMDBKTLI5E.swiftmodule: Operation not permitted
/private/var/folders/5_/q4yl04gs2kld1zztqxkqjdgh0000gq/T/SwiftTemplate/1FA05B4D-851D-4D2B-ADD6-E5A0DF70CD37/2.3.0/Package.swift:4:8: error: failed to build module 'PackageDescription' for importation due to the errors above; the textual interface may be broken by project issues or a compiler bug
 2 | // The swift-tools-version declares the minimum version of Swift required to build this package.
 3 | 
 4 | import PackageDescription
   |        `- error: failed to build module 'PackageDescription' for importation due to the errors above; the textual interface may be broken by project issues or a compiler bug
 5 | 
 6 | let package = Package(

On a machine where this isn't working, if I open Xcode and build the project once, then xcodebuild will succeed. Even if the Xcode build fails because I didn't choose to trust the plugin. I first ran into this on our CI server, and confirmed the same behavior by creating a brand new user account on my Mac and reproducing there.

Oddly I am not able to recreate the failure after it is fixed by Xcode, even after deleting every conceivable SPM related cache.

This is the command I used (I've added a bunch of sandbox stuff to the command, with no effect):

xcodebuild -workspace Zinnia.xcworkspace -scheme Zinnia -disableAutomaticPackageResolution  -skipPackagePluginValidation -IDEPackageSupportDisableManifestSandbox=1 -IDEPackageSupportDisablePluginExecutionSandbox=1 OTHER_SWIFT_FLAGS='$(inherited) -disable-sandbox' clean test

Here is the output immediately prior to the error:

Prepare packages

Compile plug-in “PixiteDependencyGenerator” in package “pixitedependency”
[debug]: Compiling plugin to executable at /Users/natetemp/Library/Developer/Xcode/DerivedData/Zinnia-aqyjcevbunlgbtgcippbjzhhpyax/Build/Products/PluginExecutables/PixiteDependencyGenerator
[debug]: Using compiler /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/swiftc
[debug]: Plugin compilation output directory '/Users/natetemp/Library/Developer/Xcode/DerivedData/Zinnia-aqyjcevbunlgbtgcippbjzhhpyax/Build/Products/PluginExecutables'
[debug]: Computed hash of plugin compilation inputs: 44c01fc622391970b4c18b2a5fa100e2e0fa23e272829e41b575ef79872ec8f2


Apply build tool plug-in “PixiteDependencyGenerator” to target “Zinnia” in project “Zinnia”

/usr/bin/sandbox-exec -p "(version 1)
(deny default)
(import \"system.sb\")
(allow file-read*)
(allow process*)
(allow mach-lookup (global-name \"com.apple.lsd.mapdb\"))
(allow file-write*
    (subpath \"/private/tmp\")
    (subpath \"/private/var/folders/5_/q4yl04gs2kld1zztqxkqjdgh0000gq/T\")
)
(deny file-write*
    (subpath \"/Users/natetemp/projects/pixite/Zinnia\")
)
(allow file-write*
    (subpath \"/Users/natetemp/Library/Developer/Xcode/DerivedData/Zinnia-aqyjcevbunlgbtgcippbjzhhpyax/Build/Intermediates.noindex/BuildToolPluginIntermediates/Zinnia.output/Zinnia/PixiteDependencyGenerator\")
    (subpath \"/private/var/folders/5_/q4yl04gs2kld1zztqxkqjdgh0000gq/T/TemporaryItems\")
)
" /Users/natetemp/Library/Developer/Xcode/DerivedData/Zinnia-aqyjcevbunlgbtgcippbjzhhpyax/SourcePackages/artifacts/pixitedependency/Sourcery/sourcery-2.3.0.artifactbundle/sourcery/bin/sourcery --templates /Users/natetemp/Library/Developer/Xcode/DerivedData/Zinnia-aqyjcevbunlgbtgcippbjzhhpyax/SourcePackages/artifacts/pixitedependency/Sourcery/sourcery-2.3.0.artifactbundle/sourcery/Templates --output /Users/natetemp/Library/Developer/Xcode/DerivedData/Zinnia-aqyjcevbunlgbtgcippbjzhhpyax/Build/Intermediates.noindex/BuildToolPluginIntermediates/Zinnia.output/Zinnia/PixiteDependencyGenerator/GeneratedSources --cacheBasePath /Users/natetemp/Library/Developer/Xcode/DerivedData/Zinnia-aqyjcevbunlgbtgcippbjzhhpyax/Build/Intermediates.noindex/BuildToolPluginIntermediates/Zinnia.output/Zinnia/PixiteDependencyGenerator/Cache --sources /Users/natetemp/projects/pixite/Zinnia/Zinnia

For the record, I did ensure that the 3 directories it complains about do exist with proper permissions.

Is it possible that the sandbox being applied to the plugin is also being incorrectly applied to something involved in parsing and caching the manifest file? Or if there is something I am doing wrong, please let me know.

This is using Xcode 16.4 on macOS 15.5.