Random global network outage triggered by NEFilterDataProvider extension – only reboot helps, reinstall doesn't

I’m presuming that this is on the Mac. If not, lemme know.

it suggested the possibility that the kernel might have marked the extension as untrusted

That’s nonsense.

As to the actual cause, it’s hard to say. I’ve definitely seen similar reports, but I don’t remember the resolution.

As soon as I stop the extension … the network immediately recovers

Does that terminate your sysex process?

If it does, then that’s a strong indicating of an OS-level bug, because something in the OS is holding on to the bad state that causes the problem to come back when your start your filter again.

OTOH, if it doesn’t then it’s possible that the bad state in being stored within your sysex process. In that case — and this is only as an experiment not as an actual workaround — try killing that process. Does that clear the bad state and allow your filter to function again?

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

@DTS Engineer

I’m not quite sure what you mean by the 'sysex process.' I have tried numerous ways to restore the network state without rebooting, including a clean uninstallation of my application—system extensions, the wrapper program, and all related auxiliary tools included—followed by a fresh reinstall.

I’m increasingly convinced this is a system bug. My testing shows it occurs far more frequently in version 15.6 than in 26.2. While the issue still persists in 26.2, the probability has significantly decreased. What I urgently need to know is: Is there a way to forcibly reset the entire network stack to clear out potential 'bad states' without having to restart the machine?

I’m not quite sure what you mean by the 'sysex process.'

OK, lemme clarify.

On macOS, content filters must be packaged as a system extension [1], aka a sysex. A sysex is much like a launchd daemon, in that it can potentially run for the entire boot cycle, that is, start when you boot the Mac and not stop until you shut down [2]. Your Network Extension provider instances are created within that sysex process. So, consider this sequence:

1. You boot your Mac.
2. Your sysex starts.
3. NE instantiates your content file provider within that sysex process.
4. You stop the filter.
5. NE stops the content filter provider.
6. At this point the system may or may not stop your sysex process. But let’s consider what happens if it doesn’t…
7. You start the filter again.
8. NE instantiates your content file provider again, still within the same sysex process.

So, my questions are:

• Are you seeing the sysex process stop at step 6?
• If not, if you force the sysex process to stop at step 6, do you still see the problem after step 8?

You can stop the sysex process in a couple of ways:

• Run the kill tool as root.

• Run launchctl as root:

  % sudo launchctl stop LABEL
  

  where LABEL is the sysex’s job label.

I’m increasingly convinced this is a system bug.

Yeah, I’m not disagreeing with you there, and if the problem persists when you forcibly stop the sysex process then that’ll confirm that. OTOH, if forcibly stopping the sysex process clears the issue, it’s might be an OS issue or it might be an issue in your code [3] and we can explore more debugging options.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] See TN3134 Network Extension provider deployment.

[2] The exact behaviour here varies based on the type of sysex, for example, Endpoint Security system extensions always start at boot time whereas DriverKit system extensions start when the relevant accessory is attached.

[3] For example, your code might be leaking a resource that the NE infrastructure relies on.

@DTS Engineer When I enable and disable it, I do receive the startFilter and stopFilter messages, but I no longer receive the handleNewFlow message. As soon as I receive the stopFilter message, the network returns to normal. By the way,

let loNetworkRules4 = NENetworkRule(
    remoteNetwork: NWHostEndpoint(hostname: "127.0.0.1", port: "0"),
    remotePrefix: 0,
    localNetwork: NWHostEndpoint(hostname: "127.0.0.1", port: "0"),
    localPrefix: 0,
    protocol: .any,
    direction: .any
)

A rule like this has also caused complete network outages on many versions of the system. Speaking of which, I urgently need to find a solution. So, I have two requests:

1. When this issue occurs, how can I collect system logs to help us pinpoint the specific problem (it could be a system issue or an issue with our extension)?
2. Is there a way to quickly reset the network? You mentioned earlier that there is a process similar to launchd responsible for the entire lifecycle of the system extension. Can I kill this process to achieve a network reset?

thanks

I do receive the startFilter and stopFilter messages, but I no longer receive the handleNewFlow message

Well, presumably you receive the stop call before the start call (-:

But that still doesn’t answer the question I posed upthread, and the answer to that question is critical in determining how to continue investigating this issue.

When you disable and the re-enable the filter, you receive a stop and then a start call. Does the process ID change between the two?

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

@WangZiYuan I submitted FB19734992 which has the same symptoms as you do. It first appeared in 15.6, and yes, as you mentioned much better in 26.2/3/4, but not fixed yet.

It is a serious problem and can only be fixed by:

1. Reboot
2. Removing every System Extensions (which is not always feasible)

@DTS Engineer But when I disable the content filter, the process does not stop. When I restart the content filter, it is still the original process instance. When I manually kill the original process, or even reload the new system extension through the wrapper app, the process ID definitely changes, but the issue persists. The only solution I have found so far is to restart the system.

@kunal_a Yeah, it seems this is more likely a system bug rather than an issue with writing the system extension.

But when I disable the content filter, the process does not stop.

OK. That’s not unexpected, give my understanding of how NE relates to the sysex infrastructure.

When I manually kill the original process … the process ID definitely changes, but the issue persists.

Blat! I was hoping that the bogus state would be stored in the sysex process, and thus this would clear it. Given this finding, the only path forward I see is a bug report.

I submitted FB19734992

Thanks.

I can’t go into all the details here, but the executive summary is:

• For internal reasons, we have an internal bug tracking the fix (r. 172870187).
• That fix is not in any currently released or seeded version of macOS.
• Even when that fix lands, I’m not 100% confident that it’ll fix this issue, due to the convoluted path we took to get there.
• As always, I recommend that you continue testing with new macOS betas as we seed them (we’re currently seeding 26.5b1, build 23F5043g).

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"