Invalidating kernel-cached data when isDataCacheInhibited is true

Question

Created 2w

Replies 2

Boosts 0

Participants 3

I'm working on an FSKit module for EdenFS, a source control virtual filesystem.

A teammate of mine asked about FSKit here a couple of years ago before its first public release. Since then, FSKit's feature development has made it feasible for our use case, so we're looking at FSKit again as a replacement for our current NFSv3 solution.

The cache coherency API makes FSKit especially appealing to us, since we need a way to invalidate the kernel's caches after a checkout/goto operation. In another thread, an Apple engineer described it as "designed to manage cache coherency for network file systems and other kinds of file systems where an outside actor might modify the data outside of the kernel's normal data flow," which fits this case.

As I understand it, there are two caching modes we could use:

Negotiated caching, where FSKit calls the volume's open with the requested cache mode on every open of a file that isn't already open, and the volume replies with the coherency type it grants. FSKit then calls close once all references are released. That results in two extra round trips per file access, which is measurable on a workload that touches many thousands of files.
Unilateral caching, where the volume sets isDataCacheInhibited = true, and FSKit stops calling the protocol's methods. The kernel then caches on its own, meaning those extra round trips are eliminated.

As far as I can tell, setCacheState(for:cacheMode:coherencyType:action: .revoke) is the only invalidation mechanism FSKit gives a volume. With isDataCacheInhibited = true, that call returns ENOTSUP (NSPOSIXErrorDomain code 45) and the kernel keeps serving the old contents. I reproduced this on a minimal in-memory module and filed it as FB24996000 with the example attached, tested on macOS 27.2 (26B5091g).

Is this ENOTSUP intended? The setCacheState documentation says it returns ENOTSUP for volumes that don't conform to FSVolume.DataCacheHandler, and my sample does conform. However, the isDataCacheInhibited documentation says it "instructs FSKit not to call this protocol's methods, even if the volume conforms to it". Is FSKit intentionally treating an inhibited volume as if it doesn't conform? If so, is there any other way to tell the kernel that cached data is stale in this mode? The documentation also states that the property is only read at loadResource, so we can't switch to negotiated caching just to serve a goto command either.

For a source control filesystem like ours, we'd at least need to invalidate the kernel cache at specific points (after a goto) for correctness. If per-item invalidation is unsupported in this mode on purpose, would Apple consider a way to invalidate a whole volume's cache at once? Linux FUSE added FUSE_NOTIFY_INC_EPOCH (https://lists.openwall.net/linux-kernel/2025/02/20/1523) for a similar reason, so a server can invalidate every cached lookup in one call instead of one entry at a time. Negotiated caching does work (setCacheState(.revoke) succeeds in that mode) but unilateral caching is ideal for us due to the performance improvement and simpler implementation.

Answer 1 · 2w

DTS Engineer OP

Apple

2w

So, let me start here:

Unilateral caching, where the volume sets isDataCacheInhibited = true, and FSKit stops calling the protocol's methods. The kernel then caches on its own, meaning those extra round trips are eliminated.

Here is the description from the header file:

/// A Boolean value that instructs FSKit not to call this protocol's methods, even if the volume conforms to it.
///
/// FSKit reads this value after the file system replies to the `loadResource` message.
/// Changing the returned value during the runtime of the volume has no effect.

Putting that another way, "isDataCacheInhibited" is basically a global "I don't want to deal with caching at all" switch. It's only checked once (after loadResource) and, if it's set to "true", then FSKit completely ignores the entire cache handler protocol, as if you'd never implemented it at all.

The key here is that this is a one-time choice at the mount level, not a dynamic "switch". I think it's mostly intended to support things like some SAN use cases where the same "file system" can be mounted as a "shared" (like a network file system) or "exclusive" (this is the only accessor, so external caching doesn't matter), with this switch making it easy to toggle those states.

Is this ENOTSUP intended? The setCacheState documentation says it returns ENOTSUP for volumes that don't conform to FSVolume.DataCacheHandler, and my sample does conform.

Yes. If you've set "isDataCacheInhibited=true", then the expectation is that you won't call any of the cache-related functions, since they're not going to do anything. It’s returning ENOTSUP because that seemed better than silently failing.

Negotiated caching, where FSKit calls the volume's open with the requested cache mode on every open of a file that isn't already open, and the volume replies with the coherency type it grants. FSKit then calls close once all references are released. That results in two extra round trips per file access, which is measurable on a workload that touches many thousands of files.

Just to make sure I understand, you normally set "isOpenCloseInhibited", so you're now picking up an extra open/close you wouldn't otherwise have had?

For a source control filesystem like ours, we'd at least need to invalidate the kernel cache at specific points (after a goto) for correctness. If per-item invalidation is unsupported in this mode on purpose, would Apple consider a way to invalidate a whole volume's cache at once?

The behavior itself is intentional, but I don't think we'd considered the case where cache changes ONLY ever occurred across the entire volume, which (I think) is your situation. Making this explicit, what you really want here is:

No open/close calls to your extension.
The kernel can cache however it sees fit.
A global switch that lets you alter the cache state of all files.

On that last point, would setCacheState() otherwise meet your needs if #1 stopped?

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Answer 2 · 2w

helsel OP

2w

Hi Kevin,

Thanks for the detailed response. The extra context around isDataCacheInhibited and its purpose makes sense.

Just to make sure I understand, you normally set "isOpenCloseInhibited", so you're now picking up an extra open/close you wouldn't otherwise have had?

My volume doesn't adopt OpenCloseHandler, so it never sets isOpenCloseInhibited. The calls I'm seeing are DataCacheHandler's open(_:modes:cacheMode:context:replyHandler:) and close(_:context:replyHandler:). I tried adopting OpenCloseHandler alongside DataCacheHandler with isOpenCloseInhibited = true (on 26B5086k), but I still saw those DataCacheHandler calls. Among the DataCacheHandler configurations I tried, eliminating those callbacks required isDataCacheInhibited = true.

On that last point, would setCacheState() otherwise meet your needs if #1 stopped?

Yes. After a sl goto, we know exactly which items changed, and we only need to invalidate those items once, at that point. After that, the kernel can keep caching as usual until the next sl goto. We don't need to change any item's coherency type, only drop what's already cached, so either per-item setCacheState without the open/close calls or a volume-wide invalidation would cover it. Avoiding those callbacks has a measurable performance benefit in workflows I've benchmarked, but we can't use isDataCacheInhibited = true without a way to invalidate the kernel's cache to maintain correctness after a sl goto.

Best, Genevieve