fseventsd cleanup cursor exhaustion and notification stalls on macOS 26.7.1

App & System Services Core OS
rossc719 OP
Created
Replies 8
Boosts 1
Views 774
Participants 5

Feedback ID: FB25090184

I've been tracking recurring fseventsd memory growth and slow file-change notifications on macOS 26.7.1 (25G241), with massive Bazel build trees. There seems to be a repeatable boundary inside event-table cleanup. I may be missing another reset path, but the live observations match the disassembly closely.

In fseventsd 1413.160.2, ARM64E UUID CDD7E21B-7F60-3DD4-AA03-8B36745DC0F4, cleanup at RVA 0x4604 uses a 32-bit scan cursor at table+0x14. The guard at 0x464c/0x4650 skips scanning above 0xfffffdff. Each complete pass advances by 512. Bucket indexing masks the cursor, but the stored cursor keeps increasing. Starting at zero, it stops at 0xfffffe00 after 8388607 complete passes.

Table resize resets the cursor. Two observed October 3 episodes reached exactly that value, accumulated zero-reference records, and recovered without a restart when the table doubled. The resize loop rehashes records rather than freeing them, so resumed cleanup after the reset seems to explain the recovery.

The reviewed resize limit is 262144 buckets. On October 6, after the table reached that size, the next cursor exhaustion persisted. Physical footprint grew from 26.6 MB to 2.31 GB in about six minutes, and fresh FSEvents probes reached an 8-second timeout. A bounded walk found 199944 zero-reference records among 200000 visited. The walk was partial and unsuspended, with linkage errors; that is not a whole-heap count.

Restarting only fseventsd restored advancing cleanup, approximately 5 MB footprint and millisecond probes. The client mix changed after restart, so this is not a controlled fixed-client comparison. Finder stayed running, and Desktop add/delete checks worked on both sides. An earlier stale-Desktop symptom may be a separate issue.

The boundary and table-size prediction were recorded before the live observations. The Feedback report has the timeline, graphs, counter snapshots, record walks, stack excerpts, disassembly and read-only sampler source. No daemon binary or target memory was changed. Live counter access used a temporary SIP debugging exception; a severe memory/notification failure was also captured before that exception. There is not yet a small standalone workload reproducer or a patched-build comparison.

Is the scan cursor supposed to wrap or reset independently of table growth? If this is the expected mechanism, is there another cleanup path that should keep reclaiming zero-reference records once the table can no longer resize?

Replies  8
Boosts 1
Views  774
Participants  5
DTS Engineer OP
Apple

I've been tracking recurring fseventsd memory growth and slow file-change notifications on macOS 26.7.1 (25G241), with massive Bazel build trees. There seems to be a repeatable boundary inside event-table cleanup. I may be missing another reset path, but the live observations match the disassembly closely.

What's you're role in that process? Are you one of the monitoring processes or are these system processes that are already monitoring? How fast it the Basel hierarchy growing and does it need to be actively monitored?

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware
rossc719 OP

Hi Kevin. I'm debugging this as the user of the Mac. Finder and other existing applications are FSEvents clients, and the CPU/memory problems predate the diagnostic monitoring. In fact, they were the reason I started doing it.

The monitor I added samples fseventsd once a minute. A notification test briefly starts fswatch on a small private probe directory, creates a file, measures the notification delay, and stops the listener. That listener isn't watching the Bazel trees. Then it does a cursor/record inspection by reading the daemon's memory without suspending it or writing. Probe only.

I dont have a reliable files/sec or FSEvents/sec figure for the build trees. There is substantial file creation and removal during builds; the fs_usage counts I have are syscall counts, so they wouldn't establish that rate. I haven't isolated which existing subscribers need to watch those trees or whether their watches can be narrowed. That's up to them, and I dont have any direct control there. But heavy use shouldn't really break things.

What I need is for normal file-change notifications to keep working, including Finder/Desktop updates, without the sudden memory bloat and CPU load. I'm still treating the Desktop symptom separately. My concern with the cleanup code is that, once the cursor stops, zero-reference records seem to accumulate until a resize resets it. At the maximum table size, that recovery route is no longer available and fseventsd begins accumulating memory and CPU like a runaway snowball.
AlekseiShkurin OP

I’m seeing similar symptoms on macOS 15.7.9 (24G830) with fseventsd 1400.100.1, so this may also affect an older macOS release. Details here FB25224901

Before restarting the service, diagnostics showed:

• Approximately 2.9 GiB physical footprint.

• 5,212,296 live PathStringZone allocations, occupying approximately 1.6 GiB, with only about 1% fragmentation.

• Approximately 2.8 GiB of writable memory marked swapped out.

• The active notification-thread samples predominantly reached strncmp through the interned-string lookup chain.

• A short filesystem trace recorded approximately 395 MiB of page-in requests over 5.4 seconds, almost entirely on the notification thread and backed by swap. These are summed request sizes, not unique pages or a sustained-rate measurement.

Approximately six minutes after restarting only fseventsd, its footprint was 4,577 KiB, with 1,721 PathStringZone allocations. A separate short comparison trace recorded no page-in requests.

Read-only inspection of this installed version also found the mechanism described here: a 32-bit cleanup cursor at event-table offset 0x14, advancing by 512 per complete pass, with cleanup skipped above 0xfffffdff. Starting from zero, that reaches 0xfffffe00 after 8,388,607 complete passes. Startup and successful table resizing reset the cursor; the inspected resize limit is 262,144 buckets.

Important qualification: I did not capture the affected process’s live cursor value or zero-reference record counts before restarting it. This is therefore corroborating evidence for the hypothesis, not confirmation that cursor exhaustion caused my incident. The client mix also changed after restart, so this was not a controlled fixed-client comparison.

No debugger attachment, process-memory modification, binary patching or SIP changes were used. I have prepared a privacy-minimized diagnostic summary containing the relevant aggregate measurements and code findings.

Is there any additional narrowly scoped diagnostic that would help establish whether this older version is hitting the same failure mode?

DTS Engineer OP
Apple

But heavy use shouldn't really break things.

Agreed, there are definitely bugs here as, as FSEvents shouldn't impact the system under basically "any" load.

Having said that:

What I need is for normal file-change notifications to keep working, including Finder/Desktop updates, without the sudden memory bloat and CPU load.

If you're goal is to get things working better "now", then my recommendation would be to move the Bazel hierarchy into it's own file system- for example, by using an additional APFS volume within the same volume set or by creating a DiskImages.

My guess is that at least part of whats going on is that there are multiple clients monitoring the overall file systems and differences in those client configurations are causing FSEvents to hold on to a lot more event data than it "should" and, with luck, moving the data to a separate file system will help mitigate that.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware
rossc719 OP

Thanks Kevin. I already have a workaround: restart fseventsd when it goes crazy. So "fix now" isn't that urgent. I'd like to understand the cleanup failure.

Most build output, caches, and sandboxes are already on a separate APFS filesystem. Source checkouts and one temporary output base are still internal, so that isn't a complete test of your suggestion... but it is close.

The part I dont understand is why zero-reference records accumulate once the cursor stops at 0xfffffe00, then get reclaimed after a natural resize without restarting the daemon. IIUC those records should already be eligible for cleanup. Could client configuration explain that even when the reference count is zero?

Also, one correction to my earlier description: A runaway on October 20 happened with two table doublings still available. Reaching the size cap isn't necessary for the CPU and notification delays.
emergentcap OP

Kevin, adding another case from an M4 MacBook. I’m investigating this as the user of the affected machine. The memory and swap growth keeps returning after normal restarts.

Current setup, verified October 25:

  • macOS 27.0.1 (26A434).
  • Backblaze 10.0.3.1076, Continuous backup enabled.
  • Codex desktop 26.1021.31345, build 21828, installed as ChatGPT.app.
  • Claude Code and local indexing/collection tools are also running.
  • No NAS or external filesystem is currently mounted.

The clearest observations:

  • October 17: fseventsd showed approximately 36G in top’s MEM column, total system swap was 45.5 GiB, and free disk space was down to 3.18 GiB. After a normal restart, fseventsd was approximately 6.3 MiB RSS and system swap was zero.
  • October 22: another episode reached approximately 60G MEM for fseventsd, with 67 GiB of total system swap and only about 103 MiB free disk space. Following the next boot, fseventsd was approximately 7.4 MiB RSS and swap was zero.
  • October 25: on the same boot, fseventsd grew from a rounded 27G to 29G MEM. Allocated system swap increased from 33 to 35 GiB; free space was 19.62 GiB at 16:53 UTC.

MEM and RSS are different measurements. The daemon’s MEM and compressed-memory readings also overlap; I’m not adding them together. Total system swap includes other processes, so I’m not attributing all of it to fseventsd.

I initially suspected Backblaze because of the timing. During an approximately 2.5-hour pause on October 23, its transfer/index workers exited and sampled fseventsd CPU fell to about 0.5%, but approximately 13 GiB of compressed daemon memory remained. There was no reboot before that comparison, so it does not establish whether Backblaze triggered the accumulation or merely contributed activity.

A short diagnostic sample showed notification processing more active than journal writing. A recent five-minute log window did not show an overflow or identify a responsible client. The pattern resembles the retention problem discussed here, but I have not measured the cleanup cursor or zero-reference records and cannot confirm the same mechanism.

What supported, narrowly scoped diagnostics would help distinguish excessive client event retention from a cleanup failure inside fseventsd, without disabling SIP?

  1. What should be captured while the daemon is already bloated, before restarting?
  2. How can I identify the actual event-producing paths and subscriber scopes without assuming that a running process or its working directory is responsible?
  3. Is there a macOS build with a known fix, or should a separate Feedback report reference FB25090184 and FB25224901?
Hixie OP

I ran into this as well and filed this as FB25346707.
I had an AI make a little tool to automatically kill fseventsd when the problems start, see https://github.com/Hixie/contraptions/tree/main/fseventsd-watchdog (includes an analysis of the failure, which matches rossc719's earlier account exactly).
rossc719 OP

Thanks Hixie, and everyone else adding reports. Good to see another analysis arriving at the same explanation.

One wrinkle with the watchdog: on October 20 I had high CPU and delayed notifications with the cleanup cursor stuck at 0xfffffe00, but two table doublings still available. IIUC, waiting for the "event table has grown too large" warning would miss that case. Restarting fseventsd cleared it; my monitor watches CPU and notification latency too.

Also +1 to the request for diagnostics that work with SIP enabled. What should people capture before restarting? And can Apple link FB25090184, FB25224901 and FB25346707?