All About Notarization

Good afternoon, everyone, and thanks for joining us today.

I'm Garrett, and I work on the Trust and Execution Team here at Apple and today we're here to talk All About Notarization. Here's a quick agenda for the talk. We're going to start with a brief overview of what notarization is and some of the benefits that it provides.

Then we're going to talk about the application requirements to get your software notarized. And then finally we'll run through the workflows and tools you'll need to use to notarize your software. So let's get started. What exactly is notarization? Well, it's a process that we introduced last year at WWDC to help identify and block malicious software prior to distribution. Now it's an extension of the Developer ID Program, which means that you don't need to register for anything different or use different certificates which also means that you stay in control of signing and distribution of your software just like you did before notarization was introduced.

The key to this is the Notary Service which performs automated security checks on Developer ID signed content. So let's run through a little bit about what the workflow looks like when you need to start notarizing your software for the first time.

Here's a diagram that talks a little bit about what the development workflow can look like and local development remains completely unchanged. You build and sign at your desk using your Apple Developer Certificates until you have a release candidate. At that point you sign the software with your Developer ID Certificate, and you can send a copy of it to the Apple Notary Service for notarization.

When notarization is complete and successful, the Notary Service can send back a ticket which you staple to your software prior to distribution and once it's stapled, the software is ready for distribution just like you did before.

Now it's worth calling out that this workflow didn't change at all from last year, so this is just a bit of a refresher. Now what we didn't talk about last year was what happens when someone downloads your software and uses it for the first time.

So when a user downloads your stapled software and double-click it to launch it, the gatekeeper will perform a verification.

It'll check the local ticket and it will also reach out to the Notary Service via CloudKit to check for a ticket also. As long as the ticket checks out and the ticket matches the content of your app, gatekeeper will allow the application and the user will see the normal first launch prompt.

Now I want to remind everyone that notarization is not an app review.

The Notary Service performs a set of automated security checks.

Now last year we made a goal to get most responses back from the Notary Service within an hour and it actually turns out that over the last year 99% of submissions have had an answer back within 15 minutes.

Also, the status of the Notary Service is now on Apple's public status page. So you can easily check to see if there any service problems that would cause problems. Now what are the benefits to notarization? Well, there are many of them. So I'm just going to highlight a few of them today. First the Notary Service can help prevent you from inadvertently shipping a malicious dependency.

Second, apps with a hardened runtime are more secure by default, and we'll talk a little more about that later. That can help prevent your app from being abused by attackers.

Third, users are more likely to download and try new software knowing that Apple has scanned it for known security issues. And finally, notarization also provides an audit trail of software notarized by your developer ID account that you can use to check the submission history and ensure that software hasn't been released that you didn't intend to release from your account. So that's a little bit of an overview of notarization. Now let's bring up Robert to talk about the application requirements to notarize your software.

Robert.

So to start I want to say that for any of the software that you previously distributed it doesn't have to meet any new requirements. You can submit your existing distributed software for notarization as is without change, but for new software you need to make sure that it meets a few security requirements. In particular, it has to be completely and correctly signed and it needs to adopt the hardened runtime. And by new software I mean software signed on or after June 1st of 2019.

So we're going to go into detail on what we mean by both of those things, but the correct signing and the hardened runtime. So first when you, to completely sign everything you need to sign everything. That means bundles, Macho-Os, installer packages wherever they are or whether you have Mach-Os in your installer packages, installer packages in your bundles, anywhere that they're found in any place within your product they need to be signed, they need to be signed correctly.

So to sign correctly that means you have to sign bundles, Macho-Os and code, and I'll talk more about code in a second, with your developer ID application certificate and be sure to include a secure timestamp.

For executables they need to opt into the hardened runtime. You don't need to opt into the hardened runtime for dylibs or frameworks or bundles just for executables.

For installer packages you need to sign them with your Developer ID Installer Certificate and this is different from your Developer ID Application Certificate so be careful.

Also, if you choose to sign your disk images to avoid gatekeeper path randomization, those must be signed with your Developer ID Application Certificate and include a secure timestamp. So if you're using Xcode for building your package, your software, this is easy. If you turn on automatic code signing, Xcode does all of this for you, but you have to be careful. If you use script build phases or copy build phases, those might be introducing new code into your software that Xcode doesn't know about and then you have to make sure that those get correctly signed. So I mentioned code files. So when we introduced code signing a number of years ago, we documented in the technote that these things called code places. So any files found in any of these places within their bundle are considered code by the code signing infrastructure and that means they need to have an attached signature. Mach-Os are the best for this. You can embed the signature inside of any Mach-Os that you put in these places as well as for bundles, but if you put other types of files such as JPEGs or raw binary files, those have to be signed as well, but they don't get attached signature instead the signature ends up as an extended attribute. And that means that you have to be careful when you're packaging up your code to make sure that that extended attribute stays within those. To avoid having to be too careful with that we recommend that you put anything that isn't a Macho-O or a bundle containing a Macho-O in a place other than any of these places when you're structuring your app. So to get singing right when you're doing it outside of Xcode, we recommend what we call inside-out code signing. That means you sign the most deeply nested bundle or piece of code within your app first. In this case, it would be the updater.app inside of the Sparkle framework inside of the Watching Grass Grow app and then you move up a level and sign each of the things individually. Note that when you sign the Sparkle framework by itself or the Sparkle framework that grabs the Sparkle main executable as well as signing the updater.app together. And note you need to go individually to watch Grass Grow, savergrowgrass.dylib and Watching Grass Grow Helper.

And finally, after you've signed all of those you sign everything together at the top bundle and again this will sign the main executable your bundle as indicated by your Info.plist.

Some of you use the -- Deep Flag in your custom workflows, but you need to be careful. The -- Deep Flag only looks for code in code places and in this case the Grow Grass dylib the Watching Grass Grow Saver and the updater.app wouldn't be found as code. They would be signed in as resources, but they wouldn't be signed as code and, therefore, they would be rejected by the notarization unless you took the extra steps to do the inside out signing.

And see Technote 2206 for more information on inside out signing and code places after the talk.

So once you've completely and correctly signed your bundle, your software, you have to make sure that you don't invalidate your signature. That means you should never be changing files in your bundle except during installation or update and when you update make sure what is the result of that update is correctly signed and notarized on your customer system. So now we're going to dig deeper into the hardened runtime. We introduced the hardened runtime last year at WWDC and now we're going to give a bit more detail to discuss its benefits and configuration. So the hardened runtime extends many of the system integrity protections that we have on macOS to your app. This means Runtime Code Signing Enforcement, library validation, DYLD environment variable protection and debugging protection. Note that all of these protections are owned by default and not configurable on iOS but on a macOS that are configurable via entitlements that any developer can set. So if you're using Xcode, adopting the hardened runtime is easy. Just go to the signing and capabilities tab and make sure that the runtime capability is present on your target.

Then you can select which entitlements you need to configure the hardened runtime on your project using the checkboxes provided. If you're using a custom workflow outside Xcode, you can use the codesign command to adopt the hardened runtime and to do that use the option runtime command to codesign and make sure you use the timestamp option as well to ensure that there is a secure timestamp on your application. To verify that you have adopted the hardened runtime correctly use the display option to codesign with the verbosity level of 2 and look for the runtime word in the flag section. Also note that the hardened runtime is versioned. When you sign with the hardened runtime, we record what version you were signing with so that when, if we were to add additional protections to hardened runtime in the future, we'll ensure that only the ones that your app has been tested with get applied on future systems.

So what is Runtime Code Signing Enforcement? It prevents creation of executable memory without an associated code signature within your process and it does this by ensuring first that all bytes mapped into your process match their associated code signature when they're read from disk and this includes not just executable regions of your Mach-O but also the non-executable mappings like your read-only sections.

And we prevent execution for modified memory that doesn't match its signature. So by verifying that the memory is, the memory that we're reading from disk is correct as it's coming in and making sure that we can't change it we ensure the integrity of your process as it's running.

Now one of the challenges that can come up with working with the Runtime Code Signing Enforcement is if your code uses JIT to make non-native code run fast within your app. To do this we recommend that you use the allow JIT entitlement and then use the MAP-JIT flat when allocating your Read/Write/Execute memory that you're compiling the code into. This allows us to keep the rest of the protections on all of your other memory within the system while giving you this scratch space memory to do what you need with respect to JIT.

If you can't adopt the MAP-JIT flag because you don't have source code access to your JIT engine. You can use the allow-unsigned-executive-memory entitlement. This will lower the security predictions provided by Runtime Code Signing Enforcement to just verifying that for every piece of memory that does have a code signature associated all of the bytes that you read from disk are, in fact, match that, but it allows modification to any of your memory inside your process and allows the creation of unsigned executable regions.

Another thing that we've seen some developers having challenges with is if they attempt to patch some system frameworks that they've loaded in after they have adopted the hardened runtime. We don't recommend that you do this and you should see whether any of the hardened runtime features actually make the, solve the reasons why you're doing this. But if you need to the allow-unsigned-executable-memory entitlement will do what you need to allow you to modify those memory pages that you've mapped in.

So another thing that we've seen come up with respect to Runtime Code Signing Enforcement is some people have seen crashes while they're updating their app. This is because code signatures are latched to files on first use in the kernel and that means if you modify a file that has been run and was signed, then it will no longer match the signature that's sitting in the kernel and you'll see a code signature violation. What we recommend instead is that instead of modifying existing files on disk you always create a new file with the updated changes and move the old file out of the way. This will ensure that the new file on its first use gets its code signature without causing the code signature violations that you're seeing.

So next we'll talk about library validation.

So library validation protects your app from code injection and dylib hijacking by making sure that your app only loads codesigned by your team or signed by Apple. And some of you might ask, why does it need to load codesigned by Apple? Well, remember that all the frameworks and libraries that you're loading from the operating system are Apple signed. So you have to be able to call those and load them into your process.

Note that library validation prevents the loading of unsigned and adhoc signed code. So be careful during your development process. Make sure that you use Apple development certificates rather than turning off code signing or just using adhoc signing.

So library validation can cause challenges for apps that have an in process plug-in or an ecosystem. We recommend that you consider moving to an out of process plug-in model so that you don't have to load unknown third-party code into your app but if you can't, you can use the disable-library-validation entitlement and this will allow loading of unsigned and adhoc sign plug-ins. And note you can take this by itself without taking any of the runtime code and sort enforcement related entitlements by having disable-library-validation on when the system sees that you're loading a adhoc signed or an unsigned plug-in, it will lower the security of your process to allow that because you've said you want to load unsigned plug-ins.

So next is DYLD environment variable protection. DYLD environment variables can be very useful during your development process to load debug libraries into your app while you're testing or to use libraries that are, or frameworks that you're building that are in development but aren't quite ready to be built into your app just to test them, but they can be dangerous because everything that you can do during your building and testing process an attacker can do on a customer system to take advantage of privileges or data that's available to your app.

So because of this the hardened runtime blocks these variables by default when you ship with it. If you need to use DYLD environment variables during your debugging process, you can use the get-task-allow entitlement on your debug build and note that Xcode automatically puts this on for you when you build for debug and takes it off for you automatically when you build for release. Note though that if you're using a custom workflow, the notary service in most cases doesn't accept binaries with the get-task-allow entitlement. So make sure you take this entitlement off before you ship your release build to the Notary Service.

So in a few cases, we have seen developers needed to use the DYLD environment variables when they ship their app to customers and, again, we don't recommend you do this. This can be very dangerous for taking advantage of your app on customer systems, but if you need to there is an entitlement to allow-DYLD-environment-variables which will allow these to be used and is accepted by the Notary Service. Next is debugging protection. So we all know that debuggers allow developers to inspect the state of registers and memory and modify process memory. That means they allow hackers to steal sensitive user data and inject malicious code. So by default the hardened runtime doesn't allow debugging of hardened processes, but if you need to use the debugger during your development flow, again, the get-task-allow entitlement is what you need. Along with DYLD environment variables the get-task-allow entitlements allows your app to be debugged.

But be careful if you do all of your testing with the debugger attached. This will mask some of the other hardened runtime related issues that you could run into especially around Runtime Code Signing Enforcement. Basically, once the debugger attaches, we can't force code signing enforcement anymore because debuggers like setting a breakpoint automatically change your data within your process and they would just crash immediately if we continue to enforce that. So make sure you test a release build to see what other effects the Runtime Code Signing Enforcement might have and then if you need to make a debug build without get-test-allow through Xcode, you can use the CODE-SIGN-IN-INJECT-BASE- ENTITLEMENTS=NO option in your Xcode project to get all of your debug settings except get-task-allow.

So this can be a challenge also in the plug-in ecosystem because plug-in developers need to debug their plug-ins within the app that they're going to load. So, again, we recommend considering out of process plug-in model or consider shipping a debug version to, yeah, debug version of your app to register plug-in developers so that they have the power to debug, but you don't ship that to all of your customers, but if absolutely necessary, the Notary Service will accept the combination of get-task-allow entitlement and the disable-library-validation entitlement to allow this workflow.

So we'll talk briefly about protected resource access.

So we all know that your customers use their Macs to store tons of information about their lives and the Macs have access to sensors that are security sensitive.

In order to or once you've adopted the hardened runtime, your app needs to declare its intent to access any of these protected resources.

So we mentioned all of last year, but if you need to access any of these resources you need to take the entitlement on your main bundle and then declare the usage string that is associated with the entitlement so that when your app attempts to access one of these resources, the system can provide a dialogue saying, oh, this is why I need to have access to this resource so that you can collect the user's consent.

So some recommendations a summary of this section. Take only the entitlements you need. The entitlements turn off security provided by the hardened runtime and they can be inspected by anyone looking at your app to try to see what kinds of things they can do with it once it's shipped to customers. So be careful, take only what you need, put the entitlements only on the processes that need them. If you have multiple processes, multiple executables within your app, it's unlikely that all of them need the same protections. You probably aren't doing JIT in every process. You probably aren't loading plug-ins into every process. So take only the ones you need in the processes that need them and when you're declaring resource access make sure that those entitlements are only on the main bundle of your app. Those getting inherited by any other executables within your bundles, they don't need to be all around. Just on the main bundle. Now I'm going to hand it back to Garrett to go over what you need to do to actually submit for notarization. Thanks, Robert. So now you know everything to think about while you're building and designing your application to get it ready to be notarized, but how do you actually submit it to the Notary Service? Well, let's talk a little about the notarization workflow and regardless of whether you're using Xcode or have a custom-built workflow, the rough workflow is about the same.

You submit the app for notarization, you check the status of the Notary Service. Once notarization is complete, you can staple a ticket and then when you're done you might want to verify that the stapling and notarization were successful. Before we talk a little bit more about this we should talk about what and when to submit and at a minimum you should be submitting all the software that you distribute, but really it's okay to upload more regularly so anything that runs outside of a developer's machine feel free to upload as a Notary Service. You probably don't need to upload every CI build though and anyone on the team can submit process submit software for notarization. This has changed from last year when it used to be restricted to certain roles.

So now you're ready to submit to the Notary Service. Well, if you use Xcode, it's quite easy and it's built into the archive and distribution workflow. So once you build an archive, you can open up the Xcode organizer like you see here and you can select distribute app just like you did before with Developer ID.

Select Developer ID and then use the upload option to submit a copy to the Notary Service. You'll see a progress bar during the upload and after everything is complete, you'll get dropped back into the organizer and you'll notice that the status has changed to processing. Once the Notary Service is complete, you're going to push notification to Xcode and when you come back to the organizer, you'll note that the status is changed to ready distribute and in the lower right corner, the export notarized app is now available.

When you click export notarized app, Xcode will take care of stapling the app for you and what you get is completely ready for distribution.

Now we'll talk a little more about how you can verify that yourself later because that's a shared workflow between custom workflows and Xcode.

Now, if you don't use Xcode, submitting with custom workflows is equally easy. The first thing you need to think about is what you want to submit to the Notary Service.

Now the Notary Service accepts 3 main formats; disk images, installer packages and zip archives. So if your build output is anything but one of these 3, you'll need to convert it to one of these 3 formats before you send it to the Notary Service. And remember that when you're creating a zip archive it's important to include macOS specific metadata like extended attributes. If you don't know what tools to use, there is support in ditto and Archive Utility built into the operating system.

Now one other thing to think about is if you actually have a custom installer and custom installers can be a little bit trickier if they pull down content from the Internet as part of their installation or if you use custom packaging formats.

And if you have a custom installer that does one of these things, you may need to perform 2-step notarization where you actually take all the content as it's going to arrive on disk, submit that for notarization using one of the 3 supported formats, staple it up and then you submit your custom installer app separately. So now that you know what you want to submit to the Notary Service, how do you actually do it? Well, Xcode 10 and newer contains a command line tool called altool that's generally used for interacting with the Notary Service. If you do have multiple versions of Xcode, you'll want to use Xcode select to make sure that you've selected Xcode 10 or later and then you can use altool with the notarize-app command.

With that you'll need to pass in the primary bundle ID along with the file that you want upload.

You will need to authenticate with your Apple ID, but if you look at the main page you'll see the options for using the keychain or environment variables so you don't need to always type in your password. When notarization upload is complete, you'll get a RequestUUID. This is a UUID that represents your submission, which you can turn around and use with the notarization-info command also as part of altool to check on the status of processing and this is how you can find out when your notarization is complete and what the status was.

Here's an example of a successful notarization and one important thing in here is the log file URL. Regardless of whether notarization was successful or had issues you can take a look inside of the log file URL to learn a little bit more. Now the log file URLs are not long-lived.

They only work for about a day. So really you probably want to pass around a new UID and whatever you call notarization info you'll get a new log file URL.

Here's an example JSON log from a successful processing.

Notice that the status was accepted.

Now if this had failed, the thing that you really want to look at here is the issues array.

And in a successful notarization, this will generally be empty.

But if something failed, there will be object in here and each one represents some failure during notarization. It will indicate things like which binary may not have adopted the hardened runtime or if something wasn't signed properly.

So this will be the key if anything was rejected. If it was successful, you'll probably want to look in the ticket contents especially if you do anything interesting around how you package your software. The ticket content should list every binary that was discovered by the Notary Service and, therefore, every binary whose information is included in the ticket that you'll staple.

So if you notice that anything is missing in the ticket contents, you may need to try to figure out what went wrong and try it again.

Now regardless of whether you use Xcode or altool when the Notary Service is done processing a submission, you'll get an email. Here's an example of a success email indicating that this software is ready for stapling. Which brings us to the next step.

Stapling uses a tool also built into Xcode 10 and newer called Stapler. Here you can see an example invocation of stapler with a staple command that can be used to staple directly to a package or a disk image. Now note that you can't staple directly to zip files so you'll need to unpack the zip file, staple the contents of the zip file and then you can package it back up for distribution. And note that stapling of command line tools and libraries is not supported right now even though these items still can and should be notarized.

Now, after you staple the next step is to verify that everything was successfully notarized and this step varies a little bit based on what you want to verify, but the first thing is simple. If you really just want to check that something has been stapled, you can use the Stapler tool again. Here you can see the stapler tool with the validate command to check that an item has been properly stapled, but what about if you want to verify notarization on something that you didn't put up for stapling or that you didn't staple yourself? Well, then you will want to generally use the SPCTL command, which is a tool built into macOS that runs gatekeeper assessments. And this does vary slightly based on what you want to check for notarization, but if you want to check an application bundle, you can use the SPCTL command with the assess option and the verbose output and path to the app to the application.

The source will tell you whether or not it was notarized.

Notarized Developer ID means that it was successfully notarized and if it shows anything else, that mean that it wasn't notarized. If instead you want to check an installer package for notarization, you can use the SPCTL command just like before but with the additional type option and passing that it's an install.

Again, this will show you the source and if it was successfully notarized, you'll see notarized developer ID.

Next what if you want to check notarization on a signed disk image? Well, you can use a fairly similar command except now you want to use the type open and you need to pass in the context that's listed here on the slide.

That will show you the same output as before and if it says notarized Developer ID, notarization was successful for that signed disk image.

If you want to check notarization status of anything else, you'll need to fall back to using the codesign command.

Here's an example of using the codesign command with the verify function verbose output testing a very specific requirement notarized and then a path to the binary or the thing that you're trying to check and the third line of the output will say explicit requirements satisfied indicating that the test requirement you passed in on the command line was successfully satisfied, which in this case means the binary was notarized.

If it says that the explicit requirement failed, that means the binary wasn't successfully notarized.

So that's everything about verifying notarization. I want to jump back and hit on one other usage of altool that I mentioned very early in the presentation.

Altool is also able to give you a notarization history using the notarization history command of all of the software that's been submitted for notarization on your account.

Here you can see an example of the command and an example of the output. It also accepts pagination so that you can paginate through all of the submissions that have been made.

I know that that was a lot of information to digest in one short talk, but there's a few important things that I really want you to take away from this conversation.

First, it's really important to sign your software properly using inside-out code signing. This is important not just so that gatekeeper can verify your software hasn't been tempered with but also for notarization.

Second, don't take hardened runtime entitlements that you don't need. Think about the benefit that the hardened runtime provides to your app and your users and remember that every entitlement you take reduces the security of your application. So only take those that you need.

Finally notarize and staple all the software that you distribute so that it passes gatekeeper in macOS Catalina.

Thanks for attending, and if you're interested in talking more about notarization, please come to the notarization lab that's going to be following this session immediately at 4 o'clock. Also, there are a couple of other labs running throughout the week where we can talk more about security, notarization and signing. So thank you. [ Applause ]