macOS Tahoe: IPMonitor incorrectly re-ranks interfaces causing VPN DNS leaks

Are you developing a VPN product that’s affected by this? Or are you reporting this from the perspective of someone using a VPN product created by some third-party developer?

If it’s the latter, I recommend that you discuss this with the VPN product’s developer.

If it’s the former, the first thing I’d like to do is clarify whether your VPN product is based on the Network Extension architecture? Or whether it’s using legacy VPN techniques?

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Our VPN solution has been in active development and deployment for over 15 years (since 2010), serving enterprise customers worldwide.

We do leverage Network Extensions for majority of our work flows but this issue can be easily reproducible outside VPN APP using System Configuration as we already tried with POC.

I will be creating an Apple code level support ticket and attaching the required details.

We have opened a Code Support ticket (Case ID: 17936650) to address this. Please note there is also an existing Feedback Assistant ticket regarding this issue (FB19037918

We have opened a Code Support ticket (Case ID: 17936650) to address this.

That hasn’t yet come through our system. When it does, I’ll look at it and the update this thread.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

OK, I have the info from the DTS case.

We do leverage Network Extensions for majority of our work flows

That’s not really the sort of answer I was looking for )-:

Lemme explain the background to this. Historically VPN products on macOS used a variety of ad hoc techniques to get things working. DTS uses to support such things, but with the advent of Network Extension (NE) providers that’s no longer the case. We now only support VPN products based on NE.

So, if your VPN product is solely based on NE, then that’s something we support. However, if this problem only crops up because you’re doing non-NE stuff — for example, I often see legacy VPN products modify the System Configuration dynamic store directly — then we don’t support that.

So, is your VPN product playing by these rules? And more importantly, if you configure your product to play by these rules, can you reproduce this problem?

If so, my advice is that you file your own bug about this [1], making sure to:

• Emphasise that you’re seeing this with an NE provider VPN.
• Enable NE debug logging, per the VPN (Network Extension) for macOS instructions on Bug Reporting > Profiles and Logs.
• Attach a sysdiagnose log of the problem, taken immediately after seeing the issue.
• And a rough timeline of the events.

There are a couple of other things you might do to get more traction with this issue:

• Try reproducing it with one of the built-in VPN transports. If that reproduces the issue, it makes it hard for Apple to deny responsibility (-:
• If you can’t, try creating a cut down NE provider and reproducing it there. Adding the source code to such a test NE provider to your bug report can be a great help. (Don’t add the source code to your main product. That’s likely to be way too much for us to wade through.)

If you do file a bug, please post your bug number, just for the record.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] The bug number you mentioned earlier, FB19037918, didn’t get any traction. AFAICT it was filed by one of your customers, and thus lack the code-level analysis that we expect from a developer bug. Thus it was bounced back with the suggestion that the customer speak to you.

Thanks for the prompt response. We really appreciate your detailed insights into the reported problem.

To add further to our previous observations, we developed a proof-of-concept sample code that decouples/isolates the reported issue from the VPN functionality. The POC exclusively utilized System Configuration framework and its supported APIs that are documented as officially supported in macOS Tahoe.

As per our observations, Tahoe's IPMonitor process demotes the affected network interface to a lower ranking when probe server connectivity fails. This may not be an ideal approach as other FQDNs could still be reachable. Also, we don't expect the Tahoe OS to override the attribute/configuration value that was set using the documented & supported System Configuration APIs in the sample code. https://developer.apple.com/documentation/systemconfiguration?language=objc  

Given that this issue manifests with supported APIs in a non-VPN context, we have logged a TSI (Code-level Support ticket with ID 17936650) for same. It will be really helpful if we can have a short 30-minutes call with experts at Apple. It would help to ensure that we're following best practices and determine if this represents an underlying issue that is very specific to Tahoe 


We're committed to using only supported frameworks and would appreciate guidance on the proper approach to address this interface ranking behaviour.

The POC exclusively utilized System Configuration framework

Which SC API specifically? The dynamic store, that is, SCDynamicStore? And, if so, what keys are you setting?

we don't expect the Tahoe OS to override the attribute/configuration value that was set

The expected behaviour of the dynamic store is nuanced:

• We support third-party code reading and writing custom properties.
• We support third-party code reading properties that are documented in <SystemConfiguration/SCSchemaDefinitions.h>.
• We generally don’t support third-party code writing the properties documented in <SystemConfiguration/SCSchemaDefinitions.h>. Those properties are considered to be ‘owned’ by the system.

Figure 1-3 of the System Configuration Programming Guidelines [1] shows how the system populates the Setup domain of the dynamic store from SCPreferences, how various subsystems use that setup to drive their configuration, and how those subsystems reflect their current state into the State domain. Keeping that all in sync is a challenge, and if you start changing values ‘behind the back’ of the OS then weird things will happen.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] In the Documentation Archive, but still largely relevant.