Possible deadlock between es_mute_process_events and AUTH response on the same es_client_t

We occasionally observe es_mute_process_events() called on an es_client_t while an AUTH event delivered by the same client is still outstanding.

In the problematic state, es_mute_process_events() stops returning inside IOConnectCallStructMethod. Subsequent calls to es_respond_flags_result() and a fallback es_respond_auth_result() on the same es_client_t also block in IOConnectCallStructMethod, eventually causing the AUTH deadline to be missed.

If you're able to reproduce this issue, please file a bug that includes both spindump of the point your client hung and a sysdiagnose collected after the problem, then post the bug number back here.

I've shared my thoughts on your questions below, but my intuition is that the actual failure here is different than you're describing.

Is it supported to call es_mute_process_events() before responding to an outstanding AUTH event delivered by the same es_client_t?

Yes, but that’s because your client couldn't actually prevent this if it wanted to. Event delivery is asynchronous and outside of your client’s control, so there isn't any reliable way for you to "know" whether or not a given process has outstanding auth events.

Are mute operations and es_respond_*() calls serialized internally for a single es_client_t?

There's a lock inside the user clients that serializes most es user client calls; however, the actual work of the kernel client is highly optimized and should not block for any significant period of time.

Can es_mute_process_events() wait for in-flight/outstanding AUTH processing?

Can an in-progress mute operation prevent a concurrent es_respond_*() call from completing?

I wouldn't expect these operations to ever interfere with each other. Things like process muting and the result caching happen "before" we generate the auth events your client receives, so muting a process stops future events from reaching your client without having any impact on current auth events.

Is the recommended pattern to respond to the AUTH message first and update mute state asynchronously afterward?

As I noted above, you can't actually "do" this. Your client doesn't know what events have been queued up for it, so even if you've responded to the "current" auth event, you don't have any way to guarantee another one isn't right behind it. Similarly, as far as the kernel is concerned, there's no difference between a message your app is actively examining and a message that’s queued up waiting for delivery.

Having said that...

Is the recommended pattern to respond to the AUTH message first and update mute state asynchronously afterward?

...yes, I'd consider this the recommended pattern, just not for the reason(s) you’re thinking of. Your client’s primary goal is to respond to auth messages as quickly as possible, so:

1. You'd typically respond before muting since, by definition, muting first would be "delaying" the response.

2. Whatever thread you're responding from is likely responsible for processing "future" messages, so muting on that thread is also imposing some small level of delay on the "next" event that thread will next be processing.

However, on the third hand, the actual time required for either of these calls (mute or respond) is sufficiently small that, on their own, I don't think calling them from the same thread would have any real-world impact. In any situation where it appeared to be having a real-world impact, I suspect it would almost always be a secondary contributing factor, NOT the primary cause of any issue.

Ultimately, the real justification for moving mute off of your response thread is simply that your auth thread should not be doing ANY unnecessary work, and the mute simply doesn't HAVE to be on the auth thread.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Thanks, this is helpful.

The key point we would like to clarify is different: whether the implementation-level serialization of calls made on the same es_client_t can itself stop making progress.

We have several independent termination reports showing the same stack pattern. In each occurrence, the ES client is terminated with:

code: 2
indicator: EndpointSecurity client terminated because it failed to respond to a message before its deadline

At the final process snapshot, three separate threads are simultaneously blocked in IOConnectCallStructMethod:

event-processing thread
  es_respond_flags_result
  IOConnectCallStructMethod

timer thread
  es_respond_auth_result
  IOConnectCallStructMethod

worker thread
  es_mute_process_events
  IOConnectCallStructMethod

Other parts of the process continue running and producing diagnostics, while no further ES-client diagnostic entries are emitted for approximately 6–15 seconds before the deadline termination.

You mentioned that there is a lock inside the user client that serializes most ES user-client calls. Could you please clarify whether this means serialization inside the user-space Endpoint Security library or a kernel-side IOKit user-client serialization point?

More specifically, can the kernel-side per-client serialization itself stop making progress in the following way?

1. es_mute_process_events() enters IOConnectCallStructMethod and does not return.
2. While that call remains in progress, es_respond_flags_result() and a fallback es_respond_auth_result() are called concurrently on the same es_client_t.
3. Both response calls then block in IOConnectCallStructMethod, potentially behind the first call.
4. No AUTH response completes before the message deadline, and the ES client is consequently terminated.

In other words, can an ES external method that stops returning hold or block the per-client serialization point required by subsequent response calls?

Or does the implementation guarantee that an in-progress es_mute_process_events() call cannot block the kernel-side path required by es_respond_*(), making the sequence above impossible by design?

The final process snapshots show all three calls blocked simultaneously, but they do not establish which call entered first or whether one call is holding a serialization point while the others are waiting behind it.