I'm developing a macOS product that generates verifiable audit records of media-asset movement on endpoints, for professional media-production companies. It is not an antivirus or Data Loss Prevention product; it collects operating-system file-system events and converts them into tamper-evident audit evidence and audit reports. Target users need comprehensive endpoint audit trails for compliance with industry security standards, including Motion Picture Association Trusted Partner Network assessments.
The product must reliably distinguish these operations: file copy, move, rename, and volume mount and unmount — including on external volumes.
I've reviewed existing forum guidance, including Quinn's explanation that FSEvents only signals that "something changed" rather than the exact operation, and that it is designed around Spotlight and Time Machine semantics. In my own testing I've also seen inconsistent flags across cp, Finder copy, and application saves, and frequent kFSEventStreamEventFlagMustScanSubDirs events on external drives even when nothing along the path changed.
Questions:
Given the above, for an audit product that must reliably distinguish copy vs. move vs. rename, should FSEvents be treated as structurally unsuitable, with the Endpoint Security Framework adopted instead as the primary source? For capturing volume mount and unmount operations, is the Endpoint Security Framework the recommended source, or should this be combined with Disk Arbitration? Are there long-term supported APIs recommended for this type of endpoint audit product, to ensure compatibility with future macOS releases? Any recommended documentation, WWDC sessions, or sample code for this use case would be appreciated.
For context, I'm building toward a System Extension using the Endpoint Security Framework and will file the entitlement request separately; this post is to confirm the architectural direction before committing.
Thank you.
Given the above, for an audit product that must reliably distinguish copy vs. move vs. rename, should FSEvents be treated as structurally unsuitable, with the Endpoint Security Framework adopted instead as the primary source?
The short answer is that this is a REALLY deep well of issues which don't have any single answer or solution. What you have is a collection of tools and solutions, trying to solve a set of problems and issues which are just as mixed up and confusing.
Starting with the "tools" side, the big advantage ES (EndpointSecurity) has is that the way it's hooked into the system means that whatever information it gives you is fundamentally "true". The big problem is that this also means that the message volume can be ENORMOUS and, if you're on the auth path, mistakes or delays in event handling can be absolutely catastrophic. Things are easier if you're only doing notifications, but even then the message volume is high enough that processing properly can be challenging.
However, that doesn't mean the other APIs are useless. Starting with DiskArb, I've long recommended that ES developers use DiskArb as their primary volume "management" API, with ES acting as a backstop. There are a few different reasons for that:
-
Most "normal" mount/unmount activity goes through DiskArb, so if an action bypasses DiskArb, that immediately "flags" that it's worth paying attention to.
-
DiskArb's architecture means that its activity callbacks happen "before" the event actually occurs, which means the ES system is "blind" to those actions. Putting that in more concrete terms, mount and unmount approval callbacks fire before DiskArb actually calls "mount", so when a mount is denied by DiskArb... nothing happens at the ES level, as DiskArb simply “chooses" to not call mount.
-
The DiskArb denial is a "high level" failure the system is designed to "expect", so the system handles those failures much more smoothly than it handles failures in the lower-level syscall.
...and I could keep going from there. DiskArb isn't necessarily as "reliable" (it can be bypassed by directly calling mount), but it has so many other advantages that it's almost always worth using, even when you're building on ES.
FSEvents is a slightly trickier case. Its biggest "trick" is that it can tell you when changes have happened "outside" of your app "view" (for example, when the volume was mounted on a different machine), though that may not be useful in your case. However, the other thing it provides is actually this:
...FSEvents only signals that "something changed" rather than the exact operation,
...which is actually MORE than the ES system provides, at least in the general case. The problem here is what the ES system actually "does" is provide approval/notification "hooks" attached to most/all of the critical system calls in the system. You can't "open" "write" or "clone" a file without ES knowing about it. That's all great but the problem is that you CAN read/write to a file without EVER calling read or write. I have a whole thread about this issue here but the short summary is that once a file has been opened and mapped, the ES client effectively loses any ability to control what happens to that file. The process that originally opens and maps the file can transfer that memory access to another process, at which point there's no way to really track what happens "from there". However, FSEvents will pick up those changes because all it's looking at is the change that are actually happening on the disk, not how/why/who made those changes.
All of that leads to the trickiest part of all this, which starts with a warning about this:
In my own testing, I've also seen inconsistent flags across cp, Finder copy, and application saves, and frequent kFSEventStreamEventFlagMustScanSubDirs events on external drives even when nothing along the path changed.
What you're dealing with here is a standard example of the issue described in "Inferring High-Level Semantics from Low-Level Operations" an issue you're going to be running into constantly with the kind of product. NONE of our APIs are actually going to provide the information you want in "all" cases, which means you’re inevitably going to be "guessing" about what happened based on what they tell you.
The product must reliably distinguish these operations: file copy, move, rename.
The problem here is that there's a pretty big gap between how users perceive the file system working and how it actually works. As the easy example, as far as the file system is concerned, move and rename are exactly the same operation, "rename". A "move" is just a "rename" that happens to have a different destination directory than the source directory.
https://developer.apple.com/documentation/endpointsecurity/es_event_type_auth_rename
Similarly, the user perception of the file system ignores renamex_np(...,RENAME_SWAP), which atomically "swaps" the content of two objects while leaving their core metadata intact (note that this is the recommended way we implement file saving).
However, the big killer here is "file copying" is a commonly agreed-upon user hallucination, not a file system operation. That is, "new files" (ignoring edge cases like hard links) only come into existence through one of two ways:
-
A process opens a new file and writes to it.
-
A process clones an existing file (but ONLY on APFS).
"File copying" is the name people made up for an operation that means something like "a process used the contents of one object to create a new object that looked pretty similar to the old object". Most apps choose to support this hallucination because we provide system APIs that make that easy; however, nothing requires them, and there are lots of cases where they simply won't. That's a real problem if the point of your product is to try and track implicit relationships between file system objects.
__
Kevin Elliott
DTS Engineer, CoreOS/Hardware
