Hi! We're developing a security product that uses both EndpointSecurity.framework to intercept and authorize process and file events; and NetworkExtension.framework o intercept and inspect network connections. We're occasionally seeing crashes caused by Endpoint Security timeouts. After investigating several crash reports, we believe we've identified a deadlock involving code signature verification: our Network Extension intercepts connections initiated by nsurlsessiond to retrieve OCSP/CRL data (we believe these requests are made on behalf of trustd during code signature validation). To determine which policy should be applied to an intercepted connection, our Network Extension verifies the code signature of the originating process. However, that code signature verification itself blocks while waiting for the OCSP/CRL requests to complete. Since those requests are being intercepted by our Network Extension, we end up with a circular dependency:
- A process requires code signature verification.
- Signature verification triggers OCSP/CRL network requests.
- Those requests are intercepted by our Network Extension.
- Our Network Extension attempts to verify the initiator's code signature before allowing the connection.
- That verification waits for the same OCSP/CRL requests to complete.
As a result, code signature verification becomes blocked process-wide, including verification performed while handling Endpoint Security events. Eventually, our Endpoint Security client exceeds the allowed response timeout and is terminated. We're considering bypassing interception for OCSP/CRL traffic to avoid this deadlock, but we'd like to understand whether this is the recommended or most robust approach.
Questions
- Is there a reliable way to identify network connections that are fetching OCSP or CRL data for code signature validation?
- What is the relationship between trustd and nsurlsessiond for these requests? Is there a dedicated nsurlsessiond instance serving trustd, or are these requests performed by the shared system/session-wide nsurlsessiond?
- Would it be a reasonable and future-proof approach to identify these requests by checking NEAppProxyFlow.remoteHostname (for example, ocsp.apple.com and crl.apple.com) and bypassing interception for those connections?
- Is there another recommended approach to avoid this deadlock when combining Endpoint Security and Network Extension in this way?
Any guidance or best practices would be greatly appreciated. Thank you!
1- Is there a reliable way to identify network connections that are fetching OCSP or CRL data for code signature validation?
No.
I want to make sure you’ve read Inferring High-Level Semantics from Low-Level Operations. While the specific examples given there don’t apply, there’s a general lesson to be learnt, namely that such inference tightly couples you to the implementation of a specific subsystem, and that Apple doesn’t promise to leave those implementation details unchanged.
2- What is the relationship between trustd and nsurlsessiond for these requests?
Those are all implementation details.
3- Would it be a reasonable and future-proof approach to identify these requests by checking NEAppProxyFlow.remoteHostname … ?
Reasonable? Sure.
Future proof? I can’t offer an opinion on that.
4- Is there another recommended approach to avoid this deadlock … ?
My #1 favourite approach is really easy: Don’t mess with process’s running as root. IMO security products that try to apply constraints to root inevitably run into weird problems like this.
However, my further experience is that ES vendors tend not to like that answer |-:
A more palatable option is to stay out of the way of platform binaries, as indicated by is_platform_binary. Platform binaries are those built in to the OS [1]. Unless the user has disabled SIP — and if that’s the case then, realistically, all bets are off — you can reasonably assume that the /usr/libexec/trustd platform binary is actually trustd.
Another option from outside of the code signing space is to disable networking on your trust evaluation. With code signing, however, the actual trust evaluation is done by SecCodeCheckValidity(_:_:_:). You could try to get the certificate chain from the binary and do your own trust evaluation (kSecCodeInfoCertificates), but that’s starting to sound like a lot less fun.
Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"
[1] This flag is a much stronger guarantee than the documentation would have you believe. In reality, it tells you whether the binary is in the system trust cache. I filed a bug to get the docs improved (r. 182094209).