setTunnelNetworkSettings with enforceRoutes defuncts connected sockets, even with unchanged settings

Question

Created 1w

Replies 3

Boosts 1

Participants 2

We're moving our VPN client's firewall off pf (TN3165) and, following the advice in https://developer.apple.com/forums/thread/781996, relying on enforceRoutes plus a content filter. That works well for keeping traffic in the tunnel, but it has a side effect we can't work around.

We add excluded routes at runtime (hostname-based split tunnelling), so the provider calls setTunnelNetworkSettings while connected. With enforceRoutes = true, any connected socket to a tunnelled destination (TCP, or UDP after connect()) that sends during the update gets EPIPE and stays broken until the app reopens it. It happens with identical settings too, and the EPIPE raises SIGPIPE, so apps that don't ignore it are terminated.

From a sysdiagnose, every update logs "Resetting the virtual interface", the utun's address is removed and re-added about 5 ms later, and a socket that sends in that gap is marked defunct by necp_socket_find_policy_match ("Marking socket in state 258 as defunct"). With enforceRoutes = false nothing dies, but packets for tunnelled destinations leave on the physical interface during the update. It reproduces on macOS 15.7, 26.5, 27.0 and 27.2 beta 3 (26B5101f).

I filed FB25176278 with a minimal sample (a packet tunnel provider that echoes UDP, so no server is needed) and a sysdiagnose from 27.2 beta 3. Two questions:

Is there a supported way to change excludedRoutes on a running tunnel without the interface reset?
If not, is there anything a provider can do to avoid this today, short of not updating routes while connected?

Thanks!

Answered by DTS Engineer in 911065022

Thanks for the confirmation.

I can understand how NE gets itself into this state, but the final behaviour is less than helpful so I think a bug report is called for. So, thanks for filing FB25176278.

Having said that, I’m not sure if there’s an easy way to fix this. When you set enforceRoutes the system actually catches the flow at the socket level, rather than down at the IP level. If the tunnel gets reconfigured then there’s gonna be a window where things are in an inconsistent state and hence ‘poking’ the flow at that time will result in the disconnect. Hmmmm, I guess we could do some sort of atomic NECP update… but… yeah, that’s just idle speculation on my part. The NE team needs to look at this properly, and that’s what your bug report is for.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Answer 1 · 1w

DTS Engineer OP

Apple

1w

Just to clarify, this problem only occurs if the socket generates traffic during the update, that is, between when you call setTunnelNetworkSettings(_:completionHandler:) and when the completion handler runs (roughly). So, a socket that’s dormant during that time but generates traffic after the configuration has completed doesn’t have the problem.

Is that right?

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Answer 2 · 5d

rhett-cf OP

5d

Yes, that matches what we see. A socket that stays idle through the update and only sends afterwards has always survived for us, and every kill we timestamped happened while the update was in progress.

One nuance -- sending during the update doesn't guarantee a kill. Only some of the sockets that were sending at the time died. This varied between test runs but averaged around 25% of sockets being killed but up to 60%.

Apologies for the delayed response -- my notifications were setup incorrectly so didn't see your response til I manually checked.

Answer 3 · 4d

DTS Engineer OP

Apple

4d

Recommended

Thanks for the confirmation.

I can understand how NE gets itself into this state, but the final behaviour is less than helpful so I think a bug report is called for. So, thanks for filing FB25176278.

Having said that, I’m not sure if there’s an easy way to fix this. When you set enforceRoutes the system actually catches the flow at the socket level, rather than down at the IP level. If the tunnel gets reconfigured then there’s gonna be a window where things are in an inconsistent state and hence ‘poking’ the flow at that time will result in the disconnect. Hmmmm, I guess we could do some sort of atomic NECP update… but… yeah, that’s just idle speculation on my part. The NE team needs to look at this properly, and that’s what your bug report is for.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"