The notarized custom PAM module cannot function properly after unlock from screensaver

We are developing a custom login service including custom PAM module. While it initially works correctly after installation on macOS, it becomes blocked by the system during privilege escalation (coreauthd) after unlocking the screensaver.

The custom PAM module has been signed with a Developer ID certificate and submitted for notarization using notarytool.

And passed staple check.

spctl -at open --context context:primary-signature -v
pam_custom.so: accepted
source=Notarized Developer ID

Here are the detail steps:

  1. Install the custom PAM file under /usr/local/lib/pam/.
  2. Ensure SIP (System Integrity Protection) is enabled.
  3. Log in to the Mac using the custom login service developed with the custom PAM module.
  4. Successfully log in.
  5. Open System Settings > Touch ID & Password.
  6. Click Add Fingerprint to trigger local authentication. This local authentication can be passed using Touch ID without invoking the custom PAM.
  7. Cancel adding fingerprints.
  8. Allow the Mac to idle until the screensaver is triggered.
  9. Unlock the screen using the custom PAM.
  10. Go to System Settings > Touch ID & Password and click Add Fingerprint again.
  11. This time, local authentication triggers the custom PAM, but it always fails. The system log shows that the custom PAM is not found.

Related system logs at step 9:

2024-12-25 19:05:05.320264-0800 0x42f3 Error 0x0 0 0 kernel: (AppleMobileFileIntegrity) Library Validation failed: Rejecting '/usr/local/lib/pam/pam_custom.so' (Team ID: none, platform: no) for process 'authorizationhos(941)' (Team ID: N/A, platform: yes), reason: mapped file has no cdhash, completely unsigned? Code has to be at least ad-hoc signed.
2024-12-25 19:05:05.320629-0800 0x42f3 Error 0x0 0 0 kernel: (AppleMobileFileIntegrity) Library Validation failed: Rejecting '/usr/local/lib/pam/pam_custom.so' (Team ID: none, platform: no) for process 'authorizationhos(941)' (Team ID: N/A, platform: yes), reason: mapped file has no cdhash, completely unsigned? Code has to be at least
ad-hoc signed.

Related system logs at step 11:

2024-12-25 19:05:22.510658-0800 0x41a6 Error 0x0 0 0 kernel: (AppleMobileFileIntegrity) Library Validation failed: Rejecting '/usr/local/lib/pam/pam_custom.so' (Team ID: none, platform: no) for process 'coreauthd(653)' (Team ID: N/A, platform: yes), reason: mapped file has no cdhash, completely unsigned? Code has to be at least ad-hoc signed.
2024-12-25 19:05:22.510953-0800 0x41a6 Error 0x0 0 0 kernel: (AppleMobileFileIntegrity) Library Validation failed: Rejecting '/usr/local/lib/pam/pam_custom.so' (Team ID: none, platform: no) for process 'coreauthd(653)' (Team ID: N/A, platform: yes), reason: mapped file has no cdhash, completely unsigned? Code has to be at least ad-hoc signed.
2024-12-25 19:05:22.511624-0800 0x41a6 Default 0x16b99 653 0 coreauthd: (libpam.2.dylib) in openpam_load_module(): no pam_custom.so found

Why does local authentication call the custom PAM after unlocking from the screensaver?

Could this issue also be related to a code signing configuration that needs adjustment?

Sorry it’s taken me a few days to wade in here; I’m only now just catching up with the backlog that built up over the winter break.

Written by FCG in 772227021
Could this issue also be related to a code signing configuration that needs adjustment?

On your part? No.

First, some background. When you call the PAM API, PAM loads plug-ins into your process. This is subject to library validation, as explained here.

IMPORTANT The following discusses implementation details that will help you understand what’s going on, but are not considered API. Don’t ship products that rely on this stuff.

The exact rules for library validation are different for code that’s built-in to the OS [1]. For third-party code you have to opt in to library validation, either directly, via the library option when signing your code, or indirectly, via the hardened runtime. In contrast, built-in code is always subject to library validation, with an option to opt out.

You can see this in action with the authorizationhost process. It’s signed with an entitlement that allows it to disable this default library validation:

% codesign -d --entitlements - /System/Library/Frameworks/Security.framework/Versions/A/MachServices/authorizationhost.bundle
…
[Dict]
    …
    [Key] com.apple.private.security.clear-library-validation
    [Value]
        [Bool] true
    …

Contrast that with entitlements of coreauthd:

% codesign -d --entitlements - /System/Library/Frameworks/LocalAuthentication.framework/Support/coreauthd | grep -c library-validation
…
0

That’s why authorizationhost can load your PAM module but coreauthd can’t. And it’s why you can’t resolve this problem by adjusting your code signature. The gating factor here is the code signature of the Apple code.

Note The log messages might mislead you into thinking that authorizationhost is unable to load your PAM module, but that’s probably not the case. I talk about this more here.

Written by FCG in 772227021
Why does local authentication call the custom PAM after unlocking from the screensaver?

Honestly, I don’t know. However, I’m not sure we really need to know, because:

  • It definitely tries to load your module.

  • That won’t work.

  • You can’t fix that at your end.


What does your module actually do?

Because it’s possible that you can avoid this issue by switching from a PAM module to an authorisation plug-in. Authorisation plug-ins are loaded by various helper processes that expect to disable library validation.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] If you rummage around in the code signing headers (in the macOS SDK) and source code (in Darwin) you’ll see this referred to as platform code.

The notarized custom PAM module cannot function properly after unlock from screensaver
 
 
Q