XPC connection consistently invalidated on app upgrade

Question

ethanndickson OP

Created 6d

Replies 2

Boosts 0

Participants 2

Hi,

Our project is a MacOS SwiftUI GUI application that bundles a System Network Extension, signed with a Developer ID certificate for distribution outside of the app store. The system network extension is used to write a packet tunnel provider. The signing of the app & network extension is handled by XCode (v16.0.0), we do not run codesign ourselves. We have no issues with XPC or the system network extension during normal usage, nor when the application is installed on a user's device for the first time. The problem only arises when the user upgrades the application. I have experienced this issue myself, as have our users. It's been reported on Apple Silicon macbooks running at least macOS 15.3.2.

Much like the SimpleFirewall example (which we used as a reference), we use XPC for basic communication of state between the app and NE. These XPC connections stop working when the user installs a new version of the app, with OS logs from the process indicating that the connection is immediately invalidated. Subsequent connection attempts are also immediately invalidated. Toggling the VPN in system settings (or via the app) does not resolve the problem, nor does restarting the app, nor does deleting and reinstalling the app, nor does restarting the device.

The only reliable workaround is to delete the system extension in Login Items & Extensions, under Network Extensions. No device restart is necessary to garbage collect the old extension - once the extension is reapproved by the user, the XPC issue resolves itself.

This would be an acceptable workaround were it possible to automate the deleting of the system extension, but that appears deliberately not possible, and requiring our users to do this each time they update is unreasonable.

When the upgraded app is opened for the first time, the OSSystemExtensionRequest request is sent, and the outcome is that the previously installed system network extension is replaced, as both the CFBundleVersion and CFBundleShortVersionString differ. When this issue is encountered, the output of systemextensionsctl list shows the later version is installed and activated.

I've been able to reproduce this bug on my personal laptop, with SIP on and systemextensionsctl developer off, but on my work laptop with SIP off and systemextensionsctl developer on (where the network extension is replaced on each activation request, instead of only when the version strings differ), I do not encounter this issue, which leads me to believe it has something to do with the notarization process. We notarize the pkg using xcrun notarytool, and then staple to the pkg.

This is actually the same issue described in:

https://developer.apple.com/forums/thread/711713

https://developer.apple.com/forums/thread/667597

https://developer.apple.com/forums/thread/742992

https://developer.apple.com/forums/thread/728063

but it's been a while since any of these threads were updated, and we've made attempts to address it off the suggestions in the threads to no avail.

Those suggestions are:

Switching to a .pkg installer from a .dmg
As part of the .pkg preinstall, doing all of the following: Stopping the VPN (scutil --nc stop), shutting down the app (using osascript 'quit app id'), and deleting the app (which claims to delete the network extension, but not the approval in Login Items & Extensions remains??), by running rm -rf on the bundle in /Applications
As part of the .pkg postinstall: Forcing macOS to ingest the App bundle's notarization ticket using spctl --assess.
Ensuring NSXPCListener.resume() is called after autoreleasepool { NEProvider.startSystemExtensionMode() } (mentioned in a forum thread above as a fix, did not help.)

One thing I'm particularly interested in is the outcome of this feedback assistant ticket, as I can't view it: FB11086599. It was shared on this forum in the first thread above, and supposedly describes the same issue. I almost find it hard to believe that this issue has been around for this many years without a workaround (there's system network extension apps out there that appear to work fine when updating, are they not using XPC?), so I wonder if there's a fix described in that FB ticket.

Since I can't view that above feedback ticket, I've created my own: FB17032197

Boost

Answer 1

DTS Engineer OP

Apple

5d

Written by ethanndickson in 779395021

The signing of the app & network extension is handled by XCode (v16.0.0), we do not run codesign ourselves.

Really. That’s a surprise. AFAIK Xcode isn’t capable of correctly exporting a Developer ID-signed NE sysex. See Exporting a Developer ID Network Extension.

However, that’s not the main thrust of your question, so let’s continue…

Written by ethanndickson in 779395021

This is actually the same issue described in:

Right. Thanks for collecting those together. And, yeah, there’s definitely something weird going on here.

Written by ethanndickson in 779395021

One thing I'm particularly interested in is the outcome of … FB11086599

I’ll come back to that in a sec, but first:

Written by ethanndickson in 779395021

I can't view it

Yep. That’s just how Feedback Assistant works. I have some hints and tips on this front in Bug Reporting: How and Why?.

Written by ethanndickson in 779395021

I've created my own: FB17032197

Thank you!

I took a look at FB11086599 and I’m sad to say it went nowhere. Sometimes things slip through the cracks, and this is one of those cases )-:

So, I’m gonna recommend that we use your new bug to track this issue. And apropos that, I have a bunch of suggestions.

First, try to set up a reproducible case. I generally do this sort of thing in a VM. Something like:

Set up a clean VM running the latest released version of macOS (15.4 currently).
Enable NE logging, per the VPN (Network Extension) for macOS instructions on our Bug Reporting > Profiles and Logs page.
Take a snapshot, so you can revert back to this state between tests.
Install version N of your app.
Upgrade it to version N+1.
Test your XPC support.

If it works, this bug isn’t as simple as we’re suspecting, and we should talk more.

But, presuming that it fails as expected then the next step is to trigger a sysdiagnose log. Do this as soon as possible after reproducing the problem.

Then attach the sysdiagnose log and both versions of your app to your bug report.

Post back here when you’re done, because there are some things I’d like to check myself.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

1

Answer 2

ethanndickson OP

2d

That’s a surprise. AFAIK Xcode isn’t capable of correctly exporting a Developer ID-signed NE sysex

We generate our xcode project file using xcodegen, which lets us modify the entitlements (adding the system extension suffix) just before building using environment variables. Then, during the build, we also set the two different provisioning profiles using environment variables. Source available.

If it works, this bug isn’t as simple as we’re suspecting, and we should talk more.

Alas, I was not able to reproduce the issue exactly as described in a fresh VM. I'll copy-paste from the feedback assistant comment, where I attached three versions, and the sysdiagnose bundle.

There was no XPC connection invalidation when upgrading from the attached Version 2 to Version 3. Version 3 being just a few hours old. However, when upgrading from Version 2 to Version 3 on my personal macbook running macos 15.3.1, the issue occurred. In both cases, the VPN was stopped before the upgrade. Interestingly, when upgrading from Version 2 to Version 3 for a second time on the same macbook, the issue does not occur, also leading me to believe it has something to do with the ingesting of the notarization ticket.

I initially tested an upgrade from Version 1 to Version 2 on another fresh VM. What I didn't realise is that the VPN configuration applied by Version 1 had a different slug to that expected by Version 2. So, when upgrading, the VPN was not stopped by the .pkg preinstall scutil, and so Version 2 replaced the network extension whilst the VPN was running. Following this, the XPC connection was consistently invalidated between the Version 2 app, and the version 2 NE. Attached is a sysdiagnose (with NE debug logs enabled) captured right after the issue occurred.

But to sum up:

On an end-user device, an upgrade from Version 2 to 3 causes the issue. The network extension is not replaced whilst the VPN is running. This tracks with what our users are reporting. The issue only appears to occur when upgrading to Version 3 for the first time.
On a fresh VM, an upgrade from 2 to 3 does not cause the issue. I could not reproduce it in 5+ attempts.
On a fresh VM, an upgrade from 1 to 3 does cause the issue, as the network extension is replaced whilst the VPN is running. I could reproduce this consistently, but only on a fresh VM. Like on the end-user device, the issue only occurs in the VM when upgrading to Version 3 for the first time, and only if the VPN is running when the NE replacement happens.

0