FSEvents vs Endpoint Security Framework for a macOS file-operation audit product

Given the above, for an audit product that must reliably distinguish copy vs. move vs. rename, should FSEvents be treated as structurally unsuitable, with the Endpoint Security Framework adopted instead as the primary source?

The short answer is that this is a REALLY deep well of issues which don't have any single answer or solution. What you have is a collection of tools and solutions, trying to solve a set of problems and issues which are just as mixed up and confusing.

Starting with the "tools" side, the big advantage ES (EndpointSecurity) has is that the way it's hooked into the system means that whatever information it gives you is fundamentally "true". The big problem is that this also means that the message volume can be ENORMOUS and, if you're on the auth path, mistakes or delays in event handling can be absolutely catastrophic. Things are easier if you're only doing notifications, but even then the message volume is high enough that processing properly can be challenging.

However, that doesn't mean the other APIs are useless. Starting with DiskArb, I've long recommended that ES developers use DiskArb as their primary volume "management" API, with ES acting as a backstop. There are a few different reasons for that:

• Most "normal" mount/unmount activity goes through DiskArb, so if an action bypasses DiskArb, that immediately "flags" that it's worth paying attention to.

• DiskArb's architecture means that its activity callbacks happen "before" the event actually occurs, which means the ES system is "blind" to those actions. Putting that in more concrete terms, mount and unmount approval callbacks fire before DiskArb actually calls "mount", so when a mount is denied by DiskArb... nothing happens at the ES level, as DiskArb simply “chooses" to not call mount.

• The DiskArb denial is a "high level" failure the system is designed to "expect", so the system handles those failures much more smoothly than it handles failures in the lower-level syscall.

...and I could keep going from there. DiskArb isn't necessarily as "reliable" (it can be bypassed by directly calling mount), but it has so many other advantages that it's almost always worth using, even when you're building on ES.

FSEvents is a slightly trickier case. Its biggest "trick" is that it can tell you when changes have happened "outside" of your app "view" (for example, when the volume was mounted on a different machine), though that may not be useful in your case. However, the other thing it provides is actually this:

...FSEvents only signals that "something changed" rather than the exact operation,

...which is actually MORE than the ES system provides, at least in the general case. The problem here is what the ES system actually "does" is provide approval/notification "hooks" attached to most/all of the critical system calls in the system. You can't "open" "write" or "clone" a file without ES knowing about it. That's all great but the problem is that you CAN read/write to a file without EVER calling read or write. I have a whole thread about this issue here but the short summary is that once a file has been opened and mapped, the ES client effectively loses any ability to control what happens to that file. The process that originally opens and maps the file can transfer that memory access to another process, at which point there's no way to really track what happens "from there". However, FSEvents will pick up those changes because all it's looking at is the change that are actually happening on the disk, not how/why/who made those changes.

All of that leads to the trickiest part of all this, which starts with a warning about this:

In my own testing, I've also seen inconsistent flags across cp, Finder copy, and application saves, and frequent kFSEventStreamEventFlagMustScanSubDirs events on external drives even when nothing along the path changed.

What you're dealing with here is a standard example of the issue described in "Inferring High-Level Semantics from Low-Level Operations" an issue you're going to be running into constantly with the kind of product. NONE of our APIs are actually going to provide the information you want in "all" cases, which means you’re inevitably going to be "guessing" about what happened based on what they tell you.

The product must reliably distinguish these operations: file copy, move, rename.

The problem here is that there's a pretty big gap between how users perceive the file system working and how it actually works. As the easy example, as far as the file system is concerned, move and rename are exactly the same operation, "rename". A "move" is just a "rename" that happens to have a different destination directory than the source directory.

Similarly, the user perception of the file system ignores renamex_np(...,RENAME_SWAP), which atomically "swaps" the content of two objects while leaving their core metadata intact (note that this is the recommended way we implement file saving).

However, the big killer here is "file copying" is a commonly agreed-upon user hallucination, not a file system operation. That is, "new files" (ignoring edge cases like hard links) only come into existence through one of two ways:

1. A process opens a new file and writes to it.

2. A process clones an existing file (but ONLY on APFS).

"File copying" is the name people made up for an operation that means something like "a process used the contents of one object to create a new object that looked pretty similar to the old object". Most apps choose to support this hallucination because we provide system APIs that make that easy; however, nothing requires them, and there are lots of cases where they simply won't. That's a real problem if the point of your product is to try and track implicit relationships between file system objects.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Hi Kevin, this is extremely helpful — thank you for the depth here.

Two takeaways reshape our design: (1) using DiskArb as the primary volume API with ES as a backstop, and (2) treating "copy" as an inference rather than an observable operation. For an audit product, that pushes us toward separating observed facts (rename, clone, open — things ES reports as "true") from inferred semantics (copy, asset movement) that a correlation layer derives. We'll record and preserve the raw events and treat high-level interpretation as a distinct, clearly-labeled layer.

One follow-up on running ES in production at scale. Given the message volume you describe on the notify path, is there a recommended approach for keeping an ES client responsive without dropping events — for example, using es_mute_path or related APIs, early filtering of event types, or offloading correlation from the ES callback thread? I want to make sure we don't design ourselves into a position where the client falls behind and events are lost; for an audit product, a dropped event is a gap in the record.

Thanks again — this has been genuinely valuable.

Kei

Two takeaways reshape our design: (1) using DiskArb as the primary volume API with ES as a backstop, and (2) treating "copy" as an inference rather than an observable operation. For an audit product, that pushes us toward separating observed facts (rename, clone, open — things ES reports as "true") from inferred semantics (copy, asset movement) that a correlation layer derives. We'll record and preserve the raw events and treat high-level interpretation as a distinct, clearly labeled layer.

One other suggestion here— if you're focussed on monitoring/protecting specific locations/file types/etc., then the other option here is to deny access as a way to limit the "range" of possibilities. Depending on the product, this could be a hard-coded list or it could just be that the user needs to approve any app/tool before that tool is allowed to access those files.

The main advantage this offers is it reduces the “oddball" edge where a tool needs to be investigated even though the user didn't actually need/want to "do" anything with that file.

One follow-up on running ES in production at scale. Given the message volume you describe on the notify path, is there a recommended approach for keeping an ES client responsive without dropping events — for example, using es_mute_path or related APIs, early filtering of event types, or offloading correlation from the ES callback thread?

Sigh... so this is a massive topic in its own right and, in fact, the primary optimization "point" for most ES clients is all about muting paths. My first recommendation is to go read all of the endpoint security header files. The header comments are extremely high quality and should be considered the canonical documentation for the API.

In terms of the notify path, that's much lower risk than the auth path (since you can't stall the system), but the main advice is still the same:

• Don't do ANY blocking work on your receiving queue.

• You can call "es_new_client" multiple times to create multiple "client" paths, which will allow you to receive and process multiple events simultaneously.

My main concern here is that our "standard" optimization recommendation is to reduce the event volume, but that's probably NOT what you want. Putting that on concrete terms, most ES clients work by approving a syscall like "open" for a particular file and then using es_respond_auth_result(...,cache=true) so they don't see any other opens on that file. That's good, as multiple opens of the same target are VERY common. You, on the other hand, probably want to see "all" of those opens.

That leads to my other warning on this issue:

I want to make sure we don't design ourselves into a position where the client falls behind and events are lost; for an audit product, a dropped event is a gap in the record.

Keep in mind that this isn't just about handling "normal" usage, it's also about someone trying to ACTIVELY disrupt your product function. It's easy to fall into the trap of focusing on how "normal" system behavior looks, but the real risk here is outliers that push your product FAR beyond normal. Putting that in concrete terms, my guess is that the "normal" rate of open calls is somewhere in the 100/s->1000/s range, depending on usage. The problem is that I'd expect an "active" attacking process to EASILY generate many times that (10,000+?) and that's assuming it only used one process.

Finally, the best general advice I can give is on or linked off of this post, particularly this last point:

"Be your own worst enemy, particularly when it comes to testing your product."

In my experience, "real world" testing is of an ES client isn't all that useful. Unless your client is totally broken, normal usage won't generate sufficient load to actually stress your client, nor is it likely to find the kind of edge cases that routinely cause problems for shipping clients (note, this second point is less of an issue for notify-only clients). The better focus is to take the time to build testing tools that intentionally try and push your client to "destruction", then optimize whatever those tools find.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Kevin, thank you again. This gives us a much clearer direction for both the runtime architecture and the test strategy.

We will keep the initial product notify-only, perform no blocking work on the ES receive queue, move normalization and correlation to separate asynchronous processing, and build dedicated stress tools that intentionally generate extreme and adversarial event loads rather than relying only on normal-use testing.

We also appreciate your insights about access denial. For the current version, we plan to keep monitoring and enforcement separate, while retaining enforcement as a possible future mode for narrowly defined protected locations.

I have one final audit-specific question.

For a notify-only ES client, what is the recommended way to determine whether events have been missed because the client or system fell behind? Should an audit product primarily rely on the sequencing information in es_message_t, or are there other official signals or conditions that Apple recommends recording as evidence of a monitoring coverage gap?

Our goal is to avoid claiming complete coverage when the telemetry stream itself may be incomplete. Instead, we want to detect, preserve, and explicitly report any loss of monitoring coverage as part of the audit record.

Thank you so much for the detailed guidance. It has materially improved our architecture.

For a notify-only ES client, what is the recommended way to determine whether events have been missed because the client or system fell behind?

From the header doc in ESMessage.h:

 * @field seq_num Per-client, per-event-type sequence number that can be
 *        inspected to detect whether the kernel had to drop events for this
 *        client.  When no events are dropped for this client, seq_num
 *        increments by 1 for every message of that event type.  When events
 *        have been dropped, the difference between the last seen sequence
 *        number of that event type plus 1 and seq_num of the received message
 *        indicates the number of events that had to be dropped.
 *        Dropped events generally indicate that more events were generated in
 *        the kernel than the client was able to handle.

or system fell behind?

What do you mean by this? The kernel itself can't really "fall behind", as its whats doing all the "work" that you're monitoring.

Should an audit product primarily rely on the sequencing information in es_message_t, or are there other official signals or conditions that Apple recommends recording as evidence of a monitoring coverage gap?

I think that sequence number is really the only data that's useful for this, ignoring obvious things like tracking your app/processes own lifetime.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Hi Kevin,

Thank you again for the clarification.

To answer your question about the phrase “system fell behind” — that was imprecise wording on my part. What I actually meant was our own client falling behind, resulting in the kernel dropping events because our ES client could not keep up with the message rate. I wasn’t suggesting that the kernel itself was falling behind. Your explanation makes that distinction very clear, and seq_num is exactly the signal I was looking for.

This clears up what I was trying to understand and gives us a much clearer direction for how we represent monitoring coverage in our audit product. We’ll use the per-client, per-event-type sequence numbers as the authoritative indicator of event loss, while recording agent lifecycle events separately rather than trying to infer a generic “system lag.”

I really appreciate your guidance throughout these discussions. It has been extremely valuable in helping us build on Endpoint Security in a way that aligns with Apple’s design.

Best regards,

Kei

So, one follow-up here:

To answer your question about the phrase “system fell behind” — that was imprecise wording on my part. What I actually meant was our own client falling behind.

I would recommend treating your client falling behind AT ALL as an architectural issue you need to fix as part of your overall design, not an "acceptable" failure you just need to report/notify the user of. The basic issue here is that the communication "link" between the system and your client is sufficiently high performance that your client can/should always be able to receive events faster than the system can generate them.

Putting that in more concrete terms, imagine you started with the simplest possible ES client, which registers for "everything" but does NOTHING (not even logging). You're basically receiving ~one mach message per monitored action; however, the overhead of any given syscall is MUCH higher than the cost of that message. It's possible you might eventually reach the point where your thread never sleeps (because it "always" has a message to receive); however, the cost of receiving an individual mach message is sufficiently small that I suspect the rest of the system will start to fail (due to the overall system load) well before that point.

More the point, IF that was happening, the fix of your ES client side is fairly trivial— divide up the messages you're monitoring in half and use two clients instead of just one. Now you're using two threads to receive events instead of one, basically doubling your message handling rate. The bottom line here is that these issues aren't really about your client’s ability to receive events from the system; they're actually about what your client DOES with the messages it receives.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware