Notary service down again?

When the notary service actually goes down, DevForums tends to light up with complaints. I suspect that this issue in something specific to your product.

Looking at your older thread, it shows that notary responded, but with an error complaining about the format of your submission. Is that case again now?

If so, please post your notary log again and I’ll see what I can figured out.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

This is the notary log and it still contains the same critical validation errors claiming it has no signed executables or bundles. However if I skip notarization and just install it using installer -verboseR -pkg ConcealDistribution.pkg -target / it works just fine and everything installs as expected.

As I mentioned in the older thread I have not changed the structure or format of this PKG and this has been working for months up until a few weeks ago when it suddenly started failing.

{
  "logFormatVersion": 1,
  "jobId": "14cc46bd-10d5-4e9c-9ad6-86c06baad6b2",
  "status": "Invalid",
  "statusSummary": "Archive contains critical validation errors",
  "statusCode": 4000,
  "archiveFilename": "ConcealDistribution.pkg",
  "uploadDate": "2025-11-05T21:53:19.782Z",
  "sha256": "48ba80a95c818ae81cadcca07afed1ef4d0d8c120a87622b200dc9935964ee26",
  "ticketContents": null,
  "issues": [
    {
      "severity": "error",
      "code": null,
      "path": "ConcealDistribution.pkg",
      "message": "Package ConcealDistribution.pkg has no signed executables or bundles. No tickets can be generated.",
      "docUrl": null,
      "architecture": null
    },
    {
      "severity": "warning",
      "code": null,
      "path": "ConcealDistribution.pkg",
      "message": "The contents of the package at ConcealDistribution.pkg could not be extracted.",
      "docUrl": null,
      "architecture": null
    }
  ]
}

As a reminder so you don't have to flip back to the old thread this is the structure of my Distribution PKG when I perform a pkgutil --expand.

	ConcealDistribution
		├── Distribution
		├── ConcealConnect.pkg
		├── ConcealBrowse.pkg
		└── ConcealUpdate.pkg

The key error here is the could not be extracted one. The fact that the package couldn’t be extracted explains the has no signed executables or bundles failure.

You’re building this package with Apple tools, right?

If so, I’m gonna have you file a bug against the notary service. Include the notary log and a copy of the package that you submitted.

Once you’re done, reply here with the bug number and I’ll take things from there.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Apologies I am now having troubles getting this to recreate again even using the exact same submission. I will keep trying and hopefully get you something you can work with.

What do you consider building with Apple tools? This is our process for building as we are a golang application developed outside of xcode.

For each inner PKG:

1. create universal binary

2. sign and verify the binary

    codesign --sign "Developer ID Application: <my developer id>" --deep --timestamp --options=runtime <path to my binary>
   
    codesign --verify --verbose <path to my binary>
   

3. build and sign the inner pkg

    pkgbuild --root <path to input folder> ConcealBrowse.pkg --scripts <my scripts folder> --sign "Developer ID Installer: <my installer id>" --install-location /Applications --identifier io.conceal.helper --version <specific release version>
    pkgutil --check-signature signed-installers/ConcealBrowse.pkg
   

4. Notarize and staple the PKG

    xcrun notarytool submit --key <connect api key path> --key-id <my key id> -i <my api issuer> ConcealBrowse.pkg --wait
    xcrun stapler staple ConcealBrowse.pkg
   

After we repeat the above process for all three of our inner PKGs I build and sign the Distribution PKG

productbuild --distribution Distribution.xml --sign "Developer ID Installer: <my developer id>" --timestamp --package-path ConcealDistribution.pkg

Finally I notarize and staple the Distribution pkg

xcrun notarytool submit --key <connect api key path> --key-id <appstore api key id> -i <appstore api key issuer> ConcealDistribution.pkg --wait
xcrun stapler staple ConcealDistribution.pkg

When it comes to the notary process I can always notarize every single inner PKG however it is the Distribution PKG that causes problems.

So we are not building with xcode but these are all tools supported by Apple ran on Apple hardware so to me that would count but not sure if you have a different perspective on that.

I appreciate your help and as soon as I can get this to recreate again I will make sure to submit.

What do you consider building with Apple tools?

Ah, yes, I should’ve been more clear. I was referring to the tools used to build your installer package. The tools used to build the contents of the package aren’t relevant to this issue.

And it seems like you’re using Apple’s pkgbuild, pkgutil and productbuild.

I am now having troubles getting this to recreate again even using the exact same submission.

That’s so strange. The notary service is usually very consistent. If it falls over because of an unexpected input, it tends to do that reliably. But in your case the problem seems to come and go. I’ve never seen that before.

I appreciate you keeping an eye out for this happening again.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Well, this is certainly an interesting one (-:

The notary service is usually very consistent.

I asked the notary team to look at this from our side, and they saw something odd. The issue seems to be with your distribution file. In the working case it references ConcealUpdate.pkg like so:

<pkg-ref …>#ConcealUpdate.pkg</pkg-ref>

And it the failing case we see this:

<pkg-ref …>ConcealUpdate.pkg</pkg-ref>

Note that the hash (#) got dropped. It’s not clear whether this is something caused by you, or by Apple’s client side tools, or is something happening on the notary server itself, but I’m passing this info along as a thread for you to pull on.

Once you’ve had a chance to look at this, if you’re still convince it’s an issue on our side, I recommend that you file a bug (per my previous instructions).

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Thank you for following up on this and I did submit a bug report with the specific PKG submitted 78407997. However thanks to you and the notary developers I do believe I found the problem within my Distribution.xml

I went and synthesized a new Distribution.xml using productbuild --synthesize --package ConcealConnect.pkg --package ConcealUpdate.pkg --package ConcealBrowse.pkg Distribution.xml and I did notice that the result was drastically different from what I had.

The main difference is that the synthesized PKG does not by default show the choices to the user so that they can pick and choose what parts of the software distribution get installed. When I had added this capability I foolishly also added a second <pkg-ref ...> for every single PKG and this one did not contain the #.

So what I think was happening is because of this poor Distribution.xml implementation sometimes everything would work fine because the proper instantiation was first in the file and other times it would fail because the incorrect <pkg-ref ...> was being used.

For anyone else who finds this in the future this is the Distribution.xml I am using now and it seems to be working reliably.

<?xml version="1.0" encoding="utf-8"?>
<installer-gui-script minSpecVersion="1">
  <title>ConcealBrowse Installer</title>
  <options customize="always" allow-external-scripts="no"/>
  <domains enable_localSystem="true"/>

  <pkg-ref id="io.conceal.helper" auth="Root">#ConcealBrowse.pkg</pkg-ref>
  <pkg-ref id="io.conceal.updater" auth="Root">#ConcealUpdate.pkg</pkg-ref>
  <pkg-ref id="io.conceal.connect" auth="Root">#ConcealConnect.pkg</pkg-ref>

  <choices-outline>
    <line choice="browse"/>
    <line choice="updater"/>
    <line choice="connect"/>
  </choices-outline>

  <choice id="browse" visible="true" title="ConcealBrowse">
    <pkg-ref id="io.conceal.helper"/>
  </choice>

  <choice id="updater" visible="true" title="ConcealUpdate">
    <pkg-ref id="io.conceal.updater"/>
  </choice>

  <choice id="connect" visible="true" title="ConcealConnect">
    <pkg-ref id="io.conceal.connect"/>
  </choice>
</installer-gui-script>