opening a codesigned dylib with dlopen: different behaviors on Big Sur and Monterey

Hi,

We use the Thales hasplmd software licensing suite for our application. This consists of a system service (hasplmd) that lives in /usr/local/sentinel, and a dylib (haspvlib_xxxx.dylib, "xxxx" is our vendor number assigned by Thales) that we receive from a Thales server, signed by Thales, that is deposited in /private/var/hasplm, and loaded (dlopen) when hasplmd starts up.

We have a helper app that copies haspvlib_xxxx.dylib into /private/var/hasplm, and then launches the generic Thales installer to load all the hasplmd components.

We are seeing different behaviors between Big Sur (11.6.3, 20G415) and Monterey (12.2.1 21D62). Monterey dlopens the dylib without problems.

Big Sur displays a warning dialog when the Thales Installer launchctls off hasplmd at the end of installation, the dreaded "haspvlib_xxxx.dylib can't be opened because Apple cannot check it for malicious software".

The hasplmd log has a somewhat more helpful entry:

Invalid VLIB for vendor xxxx (syserr 1/dlopen(/var/hasplm/haspvlib_47332.dylib, 2): no suitable image found. Did find: /var/hasplm/haspvlib_xxxx.dylib:
 code signature in (/var/hasplm/haspvlib_xxxx.dylib) not valid for use in process using Library Validation: library load disallowed by system policy)

Running codesign on the dylib on the Big Sur machine yields:

%codesign --display --deep --verbose=3 --strict /private/var/hasplm/haspvlib_xxxx.dylib
Executable=/private/var/hasplm/haspvlib_xxxx.dylib
Identifier=com.gemalto.ldk.haspvlib.xxxx
Format=Mach-O universal (i386 x86_64)
CodeDirectory v=20500 size=6186 flags=0x10000(runtime) hashes=184+5 location=embedded
Hash type=sha256 size=32
CandidateCDHash sha1=ddfee937b5894ed4fd34c8928fd6ea11909081e5
CandidateCDHashFull sha1=ddfee937b5894ed4fd34c8928fd6ea11909081e5
CandidateCDHash sha256=71cffdb89775ff30bc5aa5401c57806664b32cda
CandidateCDHashFull sha256=71cffdb89775ff30bc5aa5401c57806664b32cda83eb38f61dad31ee63547213
Hash choices=sha1,sha256
CMSDigest=870a8c08cd0e9b0845479bf9aa548b799b9787d2b8a7ab3e772a8eb303acea28
CMSDigestType=2
CDHash=71cffdb89775ff30bc5aa5401c57806664b32cda
Signature size=9006
Authority=Developer ID Application: Gemalto, Inc. (JULFT3Z5HK)
Authority=Developer ID Certification Authority
Authority=Apple Root CA
Timestamp=Feb 24, 2022 at 3:18:45 PM
Info.plist entries=8
TeamIdentifier=JULFT3Z5HK
Runtime Version=10.13.0
Sealed Resources=none
Internal requirements count=1 size=192

The code signature for the hasplmd exec:

% codesign --display --deep --verbose=3 --strict /usr/local/sentinel/hasplmd 
Executable=/usr/local/sentinel/hasplmd
Identifier=com.gemalto.ldk.hasplmd
Format=Mach-O universal (x86_64)
CodeDirectory v=20500 size=72675 flags=0x10000(runtime) hashes=2262+5 location=embedded
Hash type=sha256 size=32
CandidateCDHash sha1=6e9b76786fd5fa1d4be89016814a8e5e50354d4f
CandidateCDHashFull sha1=6e9b76786fd5fa1d4be89016814a8e5e50354d4f
CandidateCDHash sha256=882319c9452790ba88b87a665cfd0a7cb43b8153
CandidateCDHashFull sha256=882319c9452790ba88b87a665cfd0a7cb43b8153ad8258f0e832c9b559c36057
Hash choices=sha1,sha256
CMSDigest=6d6cdffdf0011929ab6da1f31c108ee7d67f987ddf7ff8efb9ebf6f59854c0f4
CMSDigestType=2
CDHash=882319c9452790ba88b87a665cfd0a7cb43b8153
Signature size=9007
Authority=Developer ID Application: Gemalto, Inc. (JULFT3Z5HK)
Authority=Developer ID Certification Authority
Authority=Apple Root CA
Timestamp=Oct 26, 2021 at 3:56:24 AM
Info.plist entries=8
TeamIdentifier=JULFT3Z5HK
Runtime Version=10.13.0
Sealed `Resources=none`
Internal requirements count=1 size=184

Again, our Monterey machine has absolutely no problems loading the haspvlib dylib as signed. Can one of the codesigning gurus tell us why Big Sur doesn't like the signature?

Thanks Jeff Johnson Wavefunction, Inc.

Well, it appears that the  haspvlib_xxxx.dylib picked up a quarantine flag somewhere along the way, and that was causing the dlopen to fail, not a problem with code signing. Stranger yet, the same copy of the file (transferred via File Sharing) does NOT show any xattrs on Monterey. Clearing the xattrs with xattr -c on the Big Sur allowed the dylib to load.

opening a codesigned dylib with dlopen: different behaviors on Big Sur and Monterey
 
 
Q