OnDemand VPN connection stuck in NO INTERNET

We create custom VPN tunnel by overriding PacketTunnelProvider on MacOS. Normal VPN connection works seamlessly. But if we enable onDemand rules on VPN manager, intemittently during tunnel creation via OnDemand, internet goes away on machine leading to a connection stuck state.

Why does internet goes away during tunnel creation?

Answered by DTS Engineer in 876714022

OK. In that case I don’t see any way to make this work )-:

When you set an on-demand rule, connections that match that rule are held until the demand is satisfied. This makes sense when you think about the intended use case for on-demand rules, namely, a split VPN. Typically this pans out as follows:

  1. There’s a site that’s only available on the organisation’s intranet.
  2. The device manager deploys an on-demand VPN configuration to access that intranet.
  3. The user runs an app that connects to that site.
  4. The system treats that as demand and starts the VPN connection.
  5. And holds the app’s connection until the VPN connection is established.
  6. Once that’s done, it releases the app’s connection, which then connects to the site over the VPN.

This yields an obvious chicken’n’problem when the VPN provider relies on a connection that also matches the on-demand rule. The system can avoid this problems if the provider does it directly, from within its own process. This is the same sort of logic that NECP uses to avoid VPN loops. But if the provider’s connection somehow depends on some other unrelated process, tracking that dependency is hard and AFAIK there’s no facility within the system to do it.

You could file a bug about this, requesting that we tweak the system to understanding this dependency. However, that’s unlikely to be an easy fix.

Note If you do file a bug:

  • Enabled relevant debugging on a test machine (definitely VPN (Network Extension) but I think that Single Sign-On also makes sense).
  • Attach a sysdiagnose log taken from a machine in this stuck state.
  • Please post your bug number, just for the record.

As to what you can do about this right now, you need to find a way to break this dependency loop. For example:

  • You might limit the scope of your on-demand rules.
  • Or change how you authenticate these requests.
  • One switch to per-app VPN, targeting a specific list of apps that doesn’t include your SSO app.

Just as an FYI, extremely wide on-demand VPN rules are a common source of problems. I often see them deployed when folks are trying to use a packet tunnel provider for something that isn’t VPN. TN3120 Expected use cases for Network Extension packet tunnel providers has a general discussion of that issue.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

How is your provided packaged? As an app extension? Or a system extension?

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

App Extension

OK. Thanks for confirming.

When you see this, is there only one user involved? That is:

  • You’ve log in via the GUI.
  • And are working with networking, including your VPN.
  • Without logging out.
  • Or fast user switching to a different user.

Is that right?

This matters because appex packaging only really works in the context of a single user, so if you add other users into the mix things get complicated.


Also, you wrote:

internet goes away on machine leading to a connection stuck state.

Can you clarify what this means? I see two possibilities:

  • Networking goes away for other apps on the system but the system doesn’t start your packet tunnel provider.
  • Alternatively, the system starts your packet tunnel provider and it attempts to connect to your VPN server and that connection acts like there’s no Internet.

Is it either of these? Or something else?

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

  1. Single User
  2. Machine has internet -> onDemand Enabled-> calls startTunnel() -> but before even we call setTunnelNetworkSettings(), internet goes away on machine

So I am guessing some network settings are changing on machine in between leading to traffic going over incorrect interface.

Happening only when onDemand is enabled (Not at all when we manually call startTunnel and setup VPN tunnel settings).

Happening only when onDemand is enabled

OK. Then what does your on-demand setup look like?

before even we call setTunnelNetworkSettings(), internet goes away on machine

Does that mean your packet tunnel provider is able to open a network connection to the VPN server? That seems like it should be the case — most VPN clients have to contact the VPN server to get the settings they need to order to call setTunnelNetworkSettings() — but I just want to be clear.

And if that’s the case, then does the packet tunnel provider’s connect to the VPN server tear when the “internet goes away”? Or is it just that the Internet goes away for other app’s running on the Mac?

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

let rule = NEOnDemandRuleConnect() rule.interfaceTypeMatch = .any manager.onDemandRules = [rule] manager.isOnDemandEnabled = true manager.isEnabled = true manager.saveToPreferences {}

Yes, we need to contact VPN server and some other endpoints to get the configuration. But https requests are timing out because device is loosing internet connectivity (All apps + device)

It stays in NO_INTERNET state until on-demand is disabled

But https requests are timing out because device is loosing internet connectivity

Are you issuing these HTTP requests from within your packet tunnel provider process?

What API are you using? URLSession? Or something else?

And what’s this NO_INTERNET state? That’s not something I’d recognise from Apple code. Is this being returned by a library that you’re using?

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

  1. Yes, from within Packet tunnel provider process.
  2. URLSession
  3. No state as such. Device loses internet connectivity

Thanks for those answers.

1. Yes, from within Packet tunnel provider process.

OK. I’d expect that to work.

2. URLSession

If you use a low-level API to make a connection to your server, does that go through?

Specifically, I’d like you to try using Network framework for this, meaning NWConnection in Swift or nw_connection_t in C-based languages.

Note that this is just a diagnostic test, not a suggested fix. The advantage of testing this with Network framework rather than URLSession is that it’s much simpler. URLSession adds a lot of value on top of Network framework, but that also means it has a lot of additional scope for failure.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

  1. Why we are landing into no internet only during OnDemand connections? Is that a bug in Apple's ondDemand flow?

  2. Any known / documented bugs with URLSession?

Did you run the NWConnection diagnostic test I suggested in my previous post? What was the result?

Again, this test is a important because it allows us to distinguish between two cases:

  • Networking in your provider is blocked completely.
  • Networking in your provider is fine in general but there’s an issue specific to URLSession.

Knowing this is both useful as a diagnostic and, at least in the second case, raises the potential for a workaround.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

1.1.1.1 1.0.0.1

We are still trying to test with NWConnection. But we use some other auth as well where URLSession isn't used but we still land into no internet for few ms (Teams call also experiences drops). So, I believe on-demand flow itself has some issues.

But we use some other auth as well where URLSession isn't used

OK. But what API is used in that case?

we still land into no internet for few ms

Should I interpret “ms” as milliseconds?

I believe on-demand flow itself has some issues.

Oh, I’m not disagreeing with you. Rather, I’m trying to characterise this problem so that I can advise you as to how best to proceed.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

  1. Lower level cpp APIs for creating TCP socket and reading/writing over it. This works perfectly always.
  2. Yes ms is milliseconds.
  3. I think no internet is seen at app level / http clients / apis. That's the reason URLSession also fails and Teams call also experiences drops.

In this 2nd auth scenario, internet drop is only for few ms, on-demand connection succeeds eventually and we don't experience any issues.

Yes ms is milliseconds.

OK, so we’re talking about very short transient failures here, right?

If so, that’s not super unexpected. As the networking reconfigures, existing connections can fail and their replacements might not connect immediately.

Our preferred networking APIs have a waits-for-connectivity feature so, when you start a connection, it won’t fail immediately but instead will wait for the connection to start. This is very different from the traditional BSD Sockets model.

I talk about this in some depth in TN3151 Choosing the right networking API, and specifically in the Connect by name and BSD Sockets best practices sections.

This is one of the reasons why I asked how NWConnection behaves in this scenario.

One further thing to note here is that, for compatibility reasons, the waits-for-connectivity feature is not the default with URLSession. You have to enable it via the waitsForConnectivity property on your session configuration. So it’d be interesting to see how that behaves in this environment.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

waitsForConnectivity property is not enabled for URLsession configuration. URLSession leads to no internet internmittently but when it does, there is no mitigation other than disabling always-on. That's the biggest problem right now.

OK.

We’ve been talking about URLSession both inside your packet tunnel provider and inside client apps. Does this problem affect both?

And earlier I described a diagnostic test involving NWConnection. Have you run that yet? If not, please do. And once you’re done, reply back here with the results.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

  1. We are using URLSession inside startTunnel() provided by Packet tunnel provider

  2. Tested with NWConnection as well - it is also not receiving any response.

Response is received only after ondemand is disabled which brings the internet connectivity back.

I think there's an issue with onDemand flow itself rather than URLSession / NWConnection. Our 2nd test which mentioned about teams call drops also adds to it (Which doesn't use either of them)

2- Tested with NWConnection as well - it is also not receiving any response.

And just to be clear, that was from within your packet tunnel provider?

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Yes

Packet tunnel provider is caliing some another app which is using URLSession

Packet tunnel provider is caliing some another app which is using URLSession

Huh? Apps can’t ‘call’ other apps on iOS, and similarly for app extensions. So what do you mean by this?

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

OnDemand VPN connection stuck in NO INTERNET
 
 
Q