macOS 27.0/27.0.1: nstat_idle_check vs. flow-divert UDP socket close lock-order deadlock causes configd watchdog panic

Two field incidents show an _nstat_rwlock/UDP socket mutex ABBA deadlock followed by a configd watchdog panic. Raw wait-owner records and UUID-matched KDK stacks identify the same cycle in each incident's earlier configd diagnostic and later panic.

Both kernels: xnu-13432.1.9~1. Kernel UUIDs are in the attachments.

Our NEDNSProxyProvider can capture CloudShell's DNS UDP traffic and forward it via a separate upstream socket. Provider flows and CloudShell sockets have separate lifetimes.

During DNS UDP forwarding and flow/socket cleanup, CloudShell blocks in final socket close. configd's IPConfigurationAgentQueue blocks creating a UDP socket: a diagnostic is captured after 60 seconds without check-ins, followed by panic/restart after 180 seconds:

userspace watchdog timeout: no successful checkins from configd (1 induced crashes) in 180 seconds

Expected: socket close completes or returns an error. We have no deterministic reproducer, measured failure rate, or reconstructed per-flow close history.

Raw waitInfo records (mutex tokens are opaque):

Report A, 26A428:

thread 18691491: kernel mutex 0x61c831d7f38edbc3 owned by thread 19491850
thread 19491850: krwlock 0xfffffe000cb41730 for writing owned by thread 18691491
thread 19794593: krwlock 0xfffffe000cb41730 for writing owned by thread 18691491

Report B, 26A434:

thread 36984: kernel mutex 0x44aa649ef3095ae1 owned by thread 12555
thread 12555: krwlock 0xfffffe000ca01730 for writing owned by thread 36984
thread 309370: krwlock 0xfffffe000ca01730 for writing owned by thread 36984

In each incident, the first two records form a cycle in both snapshots; the third is configd. All three threads are TH_WAIT / TH_UNINT.

Caller-to-callee chains; listed offsets match in both kernels.

Kernel worker (A: TID 18691491; B: TID 36984):

nstat_idle_check + 808
  -> in_pcb_checkstate + 128
  -> udp_lock + 200
  -> lck_mtx_lock_contended + 1212

Holds _nstat_rwlock exclusively; waits for CloudShell's UDP socket mutex.

CloudShell (A: TID 19491850; B: TID 12555):

soclose + 212
  -> soclose_locked + 2300
  -> sofreelastref + 60
  -> flow_divert_detach + 332
  -> in_pcbdetach + 168
  -> nstat_pcb_detach + 92
  -> lck_rw_lock_exclusive + 180
  -> lck_rw_lock_exclusive_gen + 912

Holds the socket mutex; waits for _nstat_rwlock. XNU acquires these locks, not application code.

configd (downstream victim):

socket_common -> socreate_internal -> udp_attach
  -> nstat_udp_new_pcb -> lck_rw_lock_exclusive

Matched 27 KDK disassembly shows a provider CLOSE path through cfil_sock_is_dead that directly marks the PCB DEAD and schedules GC without nstat cleanup. Public xnu-12377.121.6 (26.5) uses ordinary detach first. Provider CLOSE and final fd close are distinct.

A DEAD PCB can retain an nstat locus/reference. Under _nstat_rwlock, nstat_idle_check removes the locus and releases the reference via in_pcb_checkstate(..., WNT_RELEASE, FALSE), which can take the socket mutex. Final close takes the reverse order; nstat_pcb_detach requests the nstat lock even when the locus is null.

This is a possible enabling path from disassembly/source analysis, not proof of the preceding provider CLOSE in either incident. The older source already has opposite static lock orders; 26.5 is not proven immune.

Our provider calls closeReadWithError:nil and closeWriteWithError:nil when finished, as documented for NEAppProxyFlow. This does not validate the entire asynchronous lifecycle.

1. Is this cycle known, and is there a fixed build?
2. How is reverse locking avoided after the direct DEAD transition?
3. Is there a supported workaround preserving flow cleanup? Close rate-limiting is not proven to eliminate the issue.
4. Which diagnostics would help correlate provider CLOSE, DEAD, and final fd close?

Four EXCERPT attachments contain watchdog messages, raw frames, KDK annotations, and wait-owner records. Full reports are retained for private Feedback Assistant submission.

• 26A428_panic_excerpt.txt
• 26A428_configd_excerpt.txt
• 26A434_panic_excerpt.txt
• 26A434_configd_excerpt.txt

2026-10-15, +0800: configd capture -> embedded panic event:

• A: 04:24:43.1258 -> 04:26:44
• B: 08:49:29.4349 -> 08:51:32

Use embedded panic times, not file/header timestamps. Original pairs match on boot identity; identifiers are omitted from excerpts.

Sources: NEAppProxyFlow, XNU 26.5 comparison.