notarytool crashes with SIGBUS mid-upload (EXC_BAD_ACCESS, stack guard fault) on nio.nioTransportServices.connectionchannel thread

xcrun notarytool submit crashes with a SIGBUS during upload, every time, for one specific build of our app. Filing this as its own post since it is a distinct, reproducible crash with a clear crash report, separate from general In Progress slowness.

The exact exception, from the macOS crash reporter, is EXC_BAD_ACCESS (SIGBUS), subtype KERN_PROTECTION_FAILURE, with the message "Could not determine thread index for stack guard region." It happens on a background thread named nio.nioTransportServices.connectionchannel, which is the SwiftNIO Network.framework transport thread handling the upload connection. The faulting stack is inside notarytool's own code, going through String.init(format:), then Foundation's NSString formatting, then CoreFoundation's CFString formatting functions, almost certainly while it is trying to format a log or progress line during the upload. The same small set of return addresses repeats many times in a row right before the crash, which looks like unbounded or very deep recursion in that logging path.

Reproduction: run xcrun notarytool submit yourbuild.zip --apple-id ... --password ... --team-id ... --wait against a roughly 180 MB zip of a signed .app. The tool prints Submission ID received, starts the S3 multipart upload, and crashes within about a second, before any upload progress is shown, with no error message. The submission is left behind on Apple's side, stuck at status In Progress, and disappears entirely about 11 to 12 hours later, at which point notarytool info for that id returns Submission does not exist or does not belong to your team.

This is completely reproducible for us: it has now happened on five separate submissions of our x64 (Intel) build, and zero times submitting our arm64 (Apple Silicon) build of the exact same app, same machine, same credentials, same day. We also tried setting the upload timeout higher with defaults write com.apple.gke.notary.tool nt-upload-connection-timeout 300, which did not help, same crash.

We have full macOS crash reports (.ips files) for two separate occurrences of this and are happy to attach or send them, along with the zip that triggers it.

Please file a bug about this, making sure to attach:

  • An example .ips crash report.
  • A copy of the submission that’s triggered the issue.
  • A sysdiagnose log taken shortly after reproducing it.

Once you’re done, reply here with number and I’ll take another look.

IMPORTANT Reply as a reply, rather than as a comment. See Quinn’s Top Ten DevForums Tips for this and other titbits.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

@DTS Engineer Here's the bug number: 24905316

Here's the bug number: FB24905316

Thanks. Looking at those crash reports it’s clear that you’re not the first person to hit this crash. See this message.

I submitted your zip archive and it worked for me. Moreover, you’re running the same tool version as I am:

% xcrun notarytool --version
1.1.3 (42)

My conclusion is that this is being triggered by something environmental. Unfortunately I don’t have enough insight into the notarytool implementation to offer any suggestions as to what that might be. I’m gonna do some digging and get back to you.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

notarytool crashes with SIGBUS mid-upload (EXC_BAD_ACCESS, stack guard fault) on nio.nioTransportServices.connectionchannel thread
 
 
Q