Hey all, hoping someone on the Metal team can take a look at this one. It's a crash
inside Metal 4's binary capture, and as far as I can tell there's no way to fix it from
the app side. Filed as FB25478288 (partial repro is link is there too).
Why I think it matters
The setup that crashes is basically the recommended Metal 4 workflow: one MTL4Compiler
with a MTL4PipelineDataSetSerializer (CaptureBinaries) attached, called from
maximumConcurrentCompilationTaskCount threads, to harvest pipelines for an offline
archive. Lots of pipelines share a vertex function, which I'd guess is true for most
real renderers.
The crash is 100% inside Metal.framework, with no app code on the crashing thread.
It's easy to hit: 3 out of 3 harvest runs crashed for us, one of them 2 s in.
The serializer is attached to the compiler implicitly, so the app can't put a lock
around whatever it does internally.
MTL4Compiler is Sendable, and the docs say it handles multithreading by default,
so I'd expect capture to be thread-safe too.
Environment
MacBook Pro M2 Max, macOS 27.2, Xcode 27.2, Metal toolchain 32023.922.4
Also reported on an M3 Pro
~33k render pipelines over ~16k vertex functions, up to ~1,000 pipelines in a row
sharing one vertex function
MTL4RenderPipelineDescriptor with MTL4LibraryFunctionDescriptor stages (sometimes
wrapped in MTL4SpecializedFunctionDescriptor), all colour attachments Unspecialized
The serializer gets written out with serializeAsArchiveAndFlushToURL: every so often,
then we start a fresh compiler + serializer
Crash
EXC_BAD_ACCESS (SIGSEGV), "possible pointer authentication failure":
objc_msgSend
MultiLevelAirntBinaryCache::getElementAtDestination(MTLBinaryKey*, ...)
-[MTLCompiler compileFunctionRequestInternal:frameworkLinking:linkDataSize:reflectionOnly:compilerTask:completionHandler:]
...
AGX::Compiler::compileProgram<AGX::VertexProgramKey, ...>It looks like a use-after-free: two compiles capturing the same vertex program at the
same time, and an entry in the destination archive gets freed while the other one is
still using it. It only happens with capture on, and it goes away if we never let two
capture compiles that share a stage function run at once.
Workaround we ship (not great)
A lock per stage function around each capture compile, plus sending runs of pipelines
that share a vertex shader to the same thread. That stopped the crashes, but we had to
put a timeout on the lock, because the compiler occasionally never returns, so in theory
it can still race.
Second, related crash
If serializeAsArchiveAndFlushToURL: runs while other threads are still compiling into
that serializer, we get a different crash (null deref at 0x30) in
dispatch_data_create_map ← -[_MTLBinaryArchive newArchiverIdWithBinaryKey:entry:srcArchiverId:functionType:].
Our standalone repro hits this one reliably. The getElementAtDestination crash only
shows up with our real workload so far.
Repro + details
https://github.com/Aeonitis/metal4-capture-race
It's a small Objective-C repro (Metal + Foundation only) with a bunch of modes, the
results so far, and a comparison with how Vulkan (VkPipelineCache) and D3D12 spell out
the thread-safety rules for this same pattern.
Questions
Is capturing from multiple threads into one MTL4PipelineDataSetSerializer meant to
be supported? If not, could the docs say so?
Is one compiler + one serializer per thread a safe workaround, or is there shared
state between compilers in the same process that would still race?
Is calling serializeAsArchiveAndFlushToURL: while compiles are still in flight on
that serializer allowed?
Is this already fixed in a seed