Issue with applying EV Code Signing Certificate on Big Sur (11.4)

Problem

I am trying to code sign our application with an EV Code Signing Certificate. When I run this command:

% codesign -s "Health Record Corporation" "MedKaz.app"

I get the following warning / error message:

Warning: unable to build chain to self-signed root for signer "Health Record Corporation"
MedKaz.app: errSecInternalComponent
%

Keychain Access

The Health Record Corporation certificate does show up under Certificates area in Keychain Access. I set the certificate to Always Trust in each category under the Trust section. However the certificate DOES NOT show up under My Certificates area. I think this is the issue which is causing the problem above (i.e. certificate should be visible in My Certificates) too.

My attempts to resolve

  • I rebooted my system and took out / put back in the secured certificate token USB drive that contains the certificate numerous times. No luck in showing up in My Certificates area.
  • I tried this recommendation to disable System Integrity Protection in Big Sur but to no avail.

Any ideas on how to resolve this problem?

OK, just to be clear, by EV you mean Extended Validation Certificate?

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Yes, EV as in Extended Validation Certificate.

Yes, EV as in Extended Validation Certificate.

OK, then I strongly recommend that you not continue down this path.

Back in the day, when code signing was introduced, Apple didn’t provide any infrastructure for issuing code signing certificates, and thus folks had to provide their own signing identities, either by generating them internally or by requesting a code signing certificate from a CA. Since then Apple has started issuing code signing certificates (via the various Apple developer programmes) and that has become the standard path. The fact that code signing still supports third-party code signing identities is now just an accident of history and thus we strongly recommend that you avoid this path in favour of using Apple-issued code signing certificates.

Specifically:

  • Non-Apple code signing identities have never been supported on iOS and friends.

  • On macOS, a non-Apple identity doesn’t buy you anything. There are two ways to distribute Mac software: The Mac App Store and Developer ID.

    • The Mac App Store has the same requirements as iOS-based platforms.

    • Non-Mac App Store code is checked by Gatekeeper, and Gatekeeper will only accept Developer ID-signed code. It treats code signed with a third-party identity the same as ad hoc signed code.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

So my understanding here is going through sectigo.com for an EV code signing certificate is not supported on the apple platform, and it is recommended to obtain an apple-issued code signing certificate. Question, can I use an apple-issued code signing certificate for both our .exe version of the application along with the .app one? Both versions are built in Eclipse using Ant. Curious if I'll have to keep the Sectigo code signing certificate for Windows purposes OR if I can use the apple-issued one for both macOSx and Windows purposes.

So my understanding here is going through sectigo.com for an EV code signing certificate is not supported on the apple platform

It’s not that it’s not supported per se, just that it doesn’t buy you very much.

and it is recommended to obtain an apple-issued code signing certificate.

Correct.

Question, can I use an apple-issued code signing certificate for both our .exe version of the application along with the .app one?

To be clear, by “our .exe version of the application” you’re referring to your Windows version, right? If so, there’s two things to consider:

  • Technical

  • Business

On the technical side, this doesn’t sound like a great idea to me but I don’t support Windows and thus can’t give you a definitive answer as to whether it’ll work or not.

On the business side, before doing anything with Apple signing certificates I strongly recommend you (well, your lawyer!) read your legal agreements with Apple. IIRC the Apple Developer Program License Agreement has something to say about this.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Ok, I joined the Apple Developer Program, created an Apple Distribution certificate for Ad Hoc distribution. The certificate was installed in Keychain Access.

When I run the command:

% codesign -s "Apple Distribution: HEALTH RECORD CORPORATION (QFX3AN8WBH)" "MedKaz.app"

I still get the following same error as I did with the Sectigo certificate:

Please advise on next steps I can try to resolve this error.

So, let’s start with some basics:

  • What platform are you targeting?

  • If it’s the Mac, are you planning to deploy via the Mac App Store? Or independently using Developer ID signing.

Also, if you’re not using Xcode then you may need to manually install the latest WWDR intermediate. See Apple Worldwide Developer Relations Intermediate Certificate.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Platform is macOS (Big Sur), independent deployment, and the application is built using Eclipse. I installed both the Apple WorldWide Developer Relations Intermediate and the Deverloper ID Application Certificates. I set both certificates to trust for all users. When I go to code sign, it works for about 30 sec or so then comes back with the same warning message and errSecInternalComponent (see screen capture). I tried to sign it again and it came back as already signed, then went to verify and it says a code object is not signed at all.

I am inferring that "most" of my application was signed and there's a code object that was not signed for whatever reason. Is there a log I can check to see what code object it is having a problem with?

I set both certificates to trust for all users.

In Keychain Access? That’s unnecessary and can cause problems. I recommend that you avoid modifying certificate trust settings for code signing certificates.

(see screen capture).

FYI, if you’re going to post a Terminal transcript it’s better to do that as a code block. I can copy text from a code block but not from a screen shot (roll on Live Text!).

When I go to code sign, it works for about 30 sec or so then comes back with the same warning message and errSecInternalComponent

The 30 second delay sounds like a network timeout. Are you always testing this on your work network? Can you retry at home, just to see if that changes things? (There are long term solutions to this but I don’t want to send you down that path without some confirmation that it’s the right one.)

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

To answer your questions:

  • I deleted the three Apple certificates I modified the trust levels on, and re-added them to KeyChain Access with keeping the trust defaults.
  • Understood on using the code block, will use that going forward.
  • I was already at home. No network issues that I know of.

Retried Attempt

I retried the code signing with keeping the trust defaults and evidently it worked this time - no error messages!! Not sure what codesign -v suppose to return, but no message when issuing that command.

robhoth@Robs-MacBook-Pro-2 MY MEDKAZ % pwd

/Volumes/MY MEDKAZ

robhoth@Robs-MacBook-Pro-2 MY MEDKAZ % codesign -s "Developer ID Application: HEALTH RECORD CORPORATION (QFX3AN8WBH)" "MedKaz.app"

robhoth@Robs-MacBook-Pro-2 MY MEDKAZ % codesign -v "MedKaz.app"                                                                   

robhoth@Robs-MacBook-Pro-2 MY MEDKAZ % 

Next Issue

The original purpose for going down the route of code signing was to get inline with Big Sur's new security model. Prior to Big Sur, our application could make a backup with no issues using our backup/recover function. With Big Sur that function stopped working. I just retied the function (with the application signed), and it still does not work. What else is there that is preventing this function from working on Big Sur? The backup algorithm is simple, it makes a copy of the application files and saves them to the user's workstation drive location of their choice, nothing more, nothing less than that.

Not sure what codesign -v suppose to return

It just exits with a status indicating whether it succeeded or not. To see more info, add more -v options. I usually use -v (to verify) and then -vvv to increase the verbosity to level 3. I’ve been reliably informed that the verbosity tops out at level 5 (-:

What else is there that is preventing this function from working on Big Sur?

I need more details to offer any insight into this. Some questions:

  • Presumably your backup function involves copying files from a source to a destination? What’s the source? And what’s the destination?

  • When things fail, what error do they fail with? It is EACCES (Permission denied, 13)? Or EPERM (Operation not permitted, 1)? See On File System Permissions for background as to why this matters.

  • Is this copy being done by code run inside your app’s process? Or by a separate helper process?

  • Is your main executable an actual executable? That is, if you look at your app’s bundle, the Info.plist should contain a CFBundleExecutable that points to an executable in Contents/MacOS. For example:

    % cat "Hex Fiend.app/Contents/Info.plist"  
    …
    <dict>
        …
        <key>CFBundleExecutable</key>
        <string>Hex Fiend</string>
        …
    </dict>
    </plist>
    % file "Hex Fiend.app/Contents/MacOS/Hex Fiend" 
    Hex Fiend.app/Contents/MacOS/Hex Fiend: Mach-O …
    

    I see a lot of Java code where the main executable is a shell script, and that will end badly if you trip over a TCC check.

  • Have you tested this on a ‘clean’ machine? Various subsystems involved here (Gatekeeper, TCC, and so on) cache information about your app and it’s not uncommon for those to get confused on development machines (because you are continuously building and rebuilding the app with different settings). I generally test this stuff on a VM, so I can get a fresh machine by restoring from a snapshot between tests. See Testing a Notarised Product for the details.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Retried -v and Tried -vvv

I am getting a different response now when I do -v and I did do -vvv. Here is what I am getting and it is after I opened the application and tried the backup function which failed. Looks like application data files were added / modified.

robhoth@Robs-MacBook-Pro-2 MY MEDKAZ % codesign -v "MedKaz.app"

MedKaz.app: a sealed resource is missing or invalid

robhoth@Robs-MacBook-Pro-2 MY MEDKAZ % codesign -vvv "MedKaz.app"

MedKaz.app: a sealed resource is missing or invalid

file added: /Volumes/MY MEDKAZ/MedKaz.app/Contents/Resources/Java/database/toBackup/medkazDB/dbex.lck

file added: /Volumes/MY MEDKAZ/MedKaz.app/Contents/Resources/Java/database/toBackup/medkazDB/db.lck

file modified: /Volumes/MY MEDKAZ/MedKaz.app/Contents/Resources/Java/database/toBackup/medkazDB/seg0/c3161.dat

file modified: /Volumes/MY MEDKAZ/MedKaz.app/Contents/Resources/Java/database/toBackup/medkazDB/seg0/c90.dat

file modified: /Volumes/MY MEDKAZ/MedKaz.app/Contents/Resources/Java/database/toBackup/medkazDB/seg0/c3150.dat

file modified: /Volumes/MY MEDKAZ/MedKaz.app/Contents/Resources/Java/database/toBackup/medkazDB/log/log98.dat

file modified: /Volumes/MY MEDKAZ/MedKaz.app/Contents/Resources/Java/database/toBackup/medkazDB/log/logmirror.ctrl

file modified: /Volumes/MY MEDKAZ/MedKaz.app/Contents/Resources/Java/database/toBackup/medkazDB/log/log.ctrl

file modified: /Volumes/MY MEDKAZ/MedKaz.app/Contents/Resources/Java/medkaz.gui.properties

file modified: /Volumes/MY MEDKAZ/MedKaz.app/Contents/Resources/Java/derby.log

file modified: /Volumes/MY MEDKAZ/MedKaz.app/Contents/Resources/Java/error.log

robhoth@Robs-MacBook-Pro-2 MY MEDKAZ % codesign -s "Developer ID Application: HEALTH RECORD CORPORATION (QFX3AN8WBH)" "MedKaz.app"

MedKaz.app: is already signed

robhoth@Robs-MacBook-Pro-2 MY MEDKAZ % 

Answers to your questions

  • Q: What’s the source? A: MedKaz.app on the USB drive. Q: And what’s the destination? A: MacBook SSD folder
  • Q: What error do they fail with? A: Previous stack trace error.
2021-04-29 23:09:41,071 ERROR com.hrc.utilities.Helper (Helper.java:1272) Could not copy from ../../../.. to /Users/matt/Desktop/Jones Merle M 04.29.2021
java.lang.NullPointerException
     at com.hrc.utilities.Helper.copyDirectory(Helper.java:1252)
     at com.hrc.common.BackupFacade.backup(BackupFacade.java:69)
     at com.hrc.ui.util.DialogDisplayHub$5.doInBackground(DialogDisplayHub.java:922)
     at com.hrc.ui.util.DialogDisplayHub$5.doInBackground(DialogDisplayHub.java:910)
     at org.jdesktop.swingworker.SwingWorker$1.call(Unknown Source)
     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
     at org.jdesktop.swingworker.SwingWorker.run(Unknown Source)
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
     at java.lang.Thread.run(Thread.java:748)
  • Q: Is this copy being done by code run inside your app’s process? A: Backup process is run inside the application with a Helper copy file class method.
  • Q: Is your main executable an actual executable? A: It points to run.sh. The application opens though and runs fine. It's just this backup / recover function that quit working. Actually started with Catalina and continues with Big Sur macOSXs.

Here is run.sh:

#!/bin/sh

APP_DIR="${0%/*}/../.."
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" > /dev/null && pwd )"

WORKING_DIR="$APP_DIR/Contents/Resources/Java"
JAVA_HOME="$APP_DIR/Contents/Resources/jre/mac8"
JAVA_EXEC="$SCRIPT_DIR/../Resources/jre/mac8/bin/java"

cd "$WORKING_DIR"
WORKING_DIR=`pwd`

exec "$JAVA_EXEC" -Djsse.enableSNIExtension=false -Xdock:name="My MedKaz" -Xdock:icon="$WORKING_DIR/../medkaz.icns" -splash:"$WORKING_DIR/../splash.gif" -Xmx1024M -jar "$WORKING_DIR/medkaz.jar"
exit 0
  • Q: Have you tested this on a ‘clean’ machine? A: Tested on the same MacBook (Big Sur 11.4) as I signed the application. I just tried on my old MacBook (Mojave 10.14.6) and the backup function works fine using the same USB drive containing our application that failed on Big Sur.
MedKaz.app: a sealed resource is missing or invalid

Oi vey! It looks like your app is modifying files inside of its own bundle. This is not supported [1] and will inevitably cause problems in the long term. You need to fix this.

java.lang.NullPointerException

Hmmm, that’s strong evidence of a bug in your code. Given the context, it’s very likely that a file system operation has failed due to some MAC restriction. Normally such a failure would be reported as a file system error, and it seems your error handling code has turned that into a null pointer access )-:

It points to run.sh.

That is going to cause problems. I encourage you to replace this with a native executable.

It's just this backup / recover function that quit working.

Right. I suspect what’s happening here is as follows:

  1. Your app starts its copy engine.

  2. That copy engine tries to access a MAC-controlled file system object (see the On File System Permissions doc I referenced recently).

  3. Normally TCC would prompt the user to grant access. This is failing here, probably because TCC is unable to find the responsible code because your app’s main executable is not native.

  4. Given the problem in the previous step, the MAC system simply fails the file system access with EPERM.

  5. Your copy engine is responding to that failure by crashing )-:

There’s a lot of speculation here, so I recommend that you try to nail down the details before taking any action. The first step in the process is to find out why your Java code is crashing with a null pointer access.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] And that’s been the case since… well… forever. This sort of thing caused problems back on the original Mac where, if you have an external floppy, it was possible to run the app from a locked floppy while booted from an unlocked one.

Your comments / My replies

  • [My Comment]: For starters, really really appreciate your insight, expertise and assistance. Want to say thank you up front!
  • [Your Comment]: "Oi vey! It looks like your app is modifying files inside of its own bundle. This is not supported [1] and will inevitably cause problems in the long term. You need to fix this." [My Reply]: Yes our application is self contained (executables, derby database, log files, etc) with the .app bundle, which we can distribute the entire application with only dealing with a single .app bundled file. This architecture has worked prior to macOSX Catalina with no issues for nearly 10 years. The backup function stopped working in Catalina and continues with Big Sur. Are you saying with the newer macOSXs, it does not support this type of "legacy" architecture (i.e. a single bundled application with executables, database, log files, etc)? If so, can you point me to an Apple architectural best practice documentation so I have some guidance on bundling an application using a derby database and logging system?
  • [Your Comment]: "Hmmm, that’s strong evidence of a bug in your code. Given the context, it’s very likely that a file system operation has failed due to some MAC restriction." [My Reply]: Again, this code worked prior to macOSX Catalina, but I infer with the newer macOSX security model (e.g. MAC restriction), it has now turned what worked before into a bug which is a null pointer exception. Is there a coding technique best practice I should use to become compliant with this new MAC restriction? btw, I'll reference your supplied link - just want to know what I am looking for.
  • [Your Comment About run.sh]: "That is going to cause problems. I encourage you to replace this with a native executable." [My Reply]: So run.sh is another "legacy" architecture technique and should be replaced with a native executable? If so, can you point me to an Apple architectural best practice documentation so I have some guidance.
  • [Your Comment]: "Right. I suspect what’s happening here is as follows:"... [My Reply]: Your five steps to a failure crash definitely sounds plausible. So for an overall remedy: (1) unbundle the application executable code with the database and log files; (2) become compliant with the new macOSX MAC restrictions using On File System Permissions; and (3) replace run.sh with a native executable. Does that three part remedy sound right? I'll definitely have to nail down a road map from the current state to a future state application architecture before diving into this work effort!

Yes our application is self contained (executables, derby database, log files, etc) with the .app bundle, which we can distribute the entire application with only dealing with a single .app bundled file.

That’s fine if you consider the database to be read only, for example, if you want it to act as a template for each new user who runs your app. Where you get into trouble is if you modify that database in place. That will break the seal on your app’s code signature and bad things will ensue.

This architecture has worked prior to macOS Catalina with no issues …

That you’ve noticed (-:

Are you saying with the newer macOSes, it does not support this type of "legacy" architecture … ?

No. I’m saying that this architecture hasn’t been supported since the original Macintosh, all the way back in 1984. The nature of unsupported things is that sometimes they work and sometimes they don’t.

If so, can you point me to an Apple architectural best practice documentation so I have some guidance on bundling an application using a derby database and logging system?

What, you mean like:

(-:

Seriously though, I’m struggling to find a modern reference that definitively states that you must not write to your own app bundle on the Mac. That’s partly because the standard documentation style focuses on what you should do rather than what you shouldn’t. You only find this sort of info when you get to DTS-authored documents, like QA1662 Why can’t I save data to my application’s bundle when running on the device?, and those tend to focus on iOS where the OS absolutely prevents self-modifying apps.

QA1662 contains a link to File System Programming Guide, which has a lot of good advice about where to store your app’s data.


Again, this code worked prior to macOS Catalina, but I infer with the newer macOS security model (e.g. MAC restriction), it has now turned what worked before into a bug which is a null pointer exception.

OK, there’s three parts to this:

  • The new MAC feature will generate file system errors in specific situations where they previously did not occur.

  • Having said that, file system errors are a fact of life and you need to handle them correctly. For example, a file system error could just as easily have been caused by the volume running out of space, or the user changing the permissions on a directory, or an actual disk error.

  • File system errors are reported by the relevant file system API. They will not generate a null pointer exception automatically [1]. The fact that your code is throwing a null pointer exception suggests that it mishandles file system errors in general, and you’re now only noticing because of the MAC change.

Is there a coding technique best practice I should use to become compliant with this new MAC restriction?

Again, this is not a new requirement. Your code should handle file system errors cleanly, regardless of their source.

As to how you do that, the goal is not to crash with a null pointer exception but rather present the user with an error alert telling them what’s gone wrong. How you do that really depends on how your code is structured, and that’s tied to your app architecture, including its third-party runtime, and thus not something that I can advise you on.

If you can boil this down to a specific Apple API, I can help you understand how to catch the errors that it generates but, honestly, I don’t think that you’ll need help with that part of this problem.

So run.sh is another "legacy" architecture technique and should be replaced with a native executable?

Yes.

And this is a new requirement. Historically macOS expected the main executable to be a native executable but using a script had no negative consequences. The advent of TCC means that this is no longer a good option. You should work with your third-party runtime vendor to see how best to adapt to this change.


Does that three part remedy sound right?

Yes.

I recommend that you start with the main executable change because I believe that fixing that will cause TCC to correctly display its MAC alert. At that point clicking Allow should let your app work, so you’ll only see the crash if you click Deny.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] This is logically impossible. Null pointer exceptions are a Java thing and Apple’s OSes have no built-in Java support.

Issue with applying EV Code Signing Certificate on Big Sur (11.4)
 
 
Q