VPN network extension not working after 15.1.1 upgrade

App & System Services Networking
sboanr OP
Created 2w
Replies 16
Boosts 0
Views 318
Participants 2

Our company has a VPN client that we develop and it works on 14.x and it was working on 15.x but ever since I have upgraded to 15.1.1, I do not see any traffic being sent to the TUN interface even though I have it configured as the default route. Can anyone provide guidance or insight into what might have changed around the Network Extensions that could have caused this? Unfortunately I cannot tell if this was happening on 15.0.1. Some things I have tried, to no avail, is disable the firewall and uninstalling/installing of the VPN client. I have no other filters installed that could be interfering. When I try and ping an address I should be able to reach, I get "no route to host" I have also used Wireshark and have observed zero traffic going to the TUN interface.

NOTE, networking works fine when the VPN client is not connected.

Answered by sboanr in 816713022

@DTS Engineer

Thank you for all your help on this. I am going to close this as answered. In debugging I found that a thread in the extension was actually throwing a BAD_INSTRUCTION error in the OpenSSL lib we were using. After upgrading to a new version of OpenSSL that fixed the BAD_INSTRUCTION everything now works. Of course the mystery is why does the same client work just fine on 14.x but 15.x it did not?
No matter it seems to be working now and that is all that matters. :-) Again thank you.

Replies  16
Boosts  0
Views  318
Participants  2
DTS Engineer OP
Apple
2w

What type of Network Extension app is this? A packet tunnel provider?

And how is it packaged? Or an appex? Or a sysex?

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

0
sboanr OP
2w

Quinn,

It is a PTP and is a sysex.

0
sboanr OP
2w

Quinn,

I was able to test 15.1.1 and our PTP client on a Intel based Mac mini and it worked successfully so either this is an issue with my M1 macbook pro or an issue with M1s in general. I will try and get some more data points. All I know is that traffic on the M1 Macbook Pro, in particular DNS requests, are not making it to the TUN device and because the requests are not getting resolved so no traffic is flowing.

0
sboanr OP
2w

@DTS Engineer

Unfortunately, I was apple to recreate the issue on another M3 MacbookPro so it does appear to only occur on Apple Silicon.

Scott

0
DTS Engineer OP
Apple
2w

So, lemme see if I understand this correctly:

  • You have two Macs, one Intel and one Apple silicon.

  • You have macOS 15.1.1 installed on both.

  • Your NE packet tunnel provider sysex works on the Intel one and fails on the Apple silicon one.

Is that right?

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

0
sboanr OP
2w

@DTS Engineer

Yes that is correct but I will retest today just to verify. I have also tried on 2 Apple Silicon macs with the same results. That said, was there any change in 15.1.1 that cause the PTP sysex to not work?

Scott

0
sboanr OP
2w

@DTS Engineer

I have again retested on the Intel based Mac Mini and it works as expected. What things can I do to even debug this beyond the wireshark trace? Is there any internal tool I can use?

Scott

0
sboanr OP
2w

@DTS Engineer

For reference here is the NEPacketTunnelNetworkSettings

Tue Dec 03 19:27:38.721 2024 UTC [0x0x16debb000] ERROR (TunNetworkInterface:268) --- {
Tue Dec 03 19:27:38.721 2024 UTC [0x0x16debb000] ERROR (TunNetworkInterface:268) ---     tunnelRemoteAddress = 172.86.175.254
Tue Dec 03 19:27:38.721 2024 UTC [0x0x16debb000] ERROR (TunNetworkInterface:268) ---     DNSSettings = {
Tue Dec 03 19:27:38.721 2024 UTC [0x0x16debb000] ERROR (TunNetworkInterface:268) ---         protocol = cleartext
Tue Dec 03 19:27:38.721 2024 UTC [0x0x16debb000] ERROR (TunNetworkInterface:268) ---         server = (
Tue Dec 03 19:27:38.721 2024 UTC [0x0x16debb000] ERROR (TunNetworkInterface:268) ---             100.127.255.254,
Tue Dec 03 19:27:38.721 2024 UTC [0x0x16debb000] ERROR (TunNetworkInterface:268) ---         )
Tue Dec 03 19:27:38.721 2024 UTC [0x0x16debb000] ERROR (TunNetworkInterface:268) ---         matchDomains = (
Tue Dec 03 19:27:38.721 2024 UTC [0x0x16debb000] ERROR (TunNetworkInterface:268) ---             ,
Tue Dec 03 19:27:38.721 2024 UTC [0x0x16debb000] ERROR (TunNetworkInterface:268) ---         )
Tue Dec 03 19:27:38.721 2024 UTC [0x0x16debb000] ERROR (TunNetworkInterface:268) ---         matchDomainsNoSearch = NO
Tue Dec 03 19:27:38.721 2024 UTC [0x0x16debb000] ERROR (TunNetworkInterface:268) ---     }
Tue Dec 03 19:27:38.721 2024 UTC [0x0x16debb000] ERROR (TunNetworkInterface:268) ---     IPv4Settings = {
Tue Dec 03 19:27:38.721 2024 UTC [0x0x16debb000] ERROR (TunNetworkInterface:268) ---         configMethod = manual
Tue Dec 03 19:27:38.721 2024 UTC [0x0x16debb000] ERROR (TunNetworkInterface:268) ---         addresses = (
Tue Dec 03 19:27:38.722 2024 UTC [0x0x16debb000] ERROR (TunNetworkInterface:268) ---             172.86.175.254,
Tue Dec 03 19:27:38.722 2024 UTC [0x0x16debb000] ERROR (TunNetworkInterface:268) ---         )
Tue Dec 03 19:27:38.722 2024 UTC [0x0x16debb000] ERROR (TunNetworkInterface:268) ---         subnetMasks = (
Tue Dec 03 19:27:38.722 2024 UTC [0x0x16debb000] ERROR (TunNetworkInterface:268) ---             255.255.255.254,
Tue Dec 03 19:27:38.722 2024 UTC [0x0x16debb000] ERROR (TunNetworkInterface:268) ---         )
Tue Dec 03 19:27:38.722 2024 UTC [0x0x16debb000] ERROR (TunNetworkInterface:268) ---         includedRoutes = (
Tue Dec 03 19:27:38.722 2024 UTC [0x0x16debb000] ERROR (TunNetworkInterface:268) ---             {
Tue Dec 03 19:27:38.722 2024 UTC [0x0x16debb000] ERROR (TunNetworkInterface:268) ---                 destinationAddress = 0.0.0.0
Tue Dec 03 19:27:38.722 2024 UTC [0x0x16debb000] ERROR (TunNetworkInterface:268) ---                 destinationSubnetMask = 0.0.0.0
Tue Dec 03 19:27:38.722 2024 UTC [0x0x16debb000] ERROR (TunNetworkInterface:268) ---             },
Tue Dec 03 19:27:38.734 2024 UTC [0x0x16debb000] ERROR (TunNetworkInterface:268) ---         )
Tue Dec 03 19:27:38.734 2024 UTC [0x0x16debb000] ERROR (TunNetworkInterface:268) ---         excludedRoutes = (
Tue Dec 03 19:27:38.734 2024 UTC [0x0x16debb000] ERROR (TunNetworkInterface:268) ---             {
Tue Dec 03 19:27:38.735 2024 UTC [0x0x16debb000] ERROR (TunNetworkInterface:268) ---                 destinationAddress = 20.253.190.7
Tue Dec 03 19:27:38.735 2024 UTC [0x0x16debb000] ERROR (TunNetworkInterface:268) ---                 destinationSubnetMask = 255.255.255.255
Tue Dec 03 19:27:38.735 2024 UTC [0x0x16debb000] ERROR (TunNetworkInterface:268) ---             },
Tue Dec 03 19:27:38.735 2024 UTC [0x0x16debb000] ERROR (TunNetworkInterface:268) ---             {
Tue Dec 03 19:27:38.735 2024 UTC [0x0x16debb000] ERROR (TunNetworkInterface:268) ---                 destinationAddress = 172.64.153.124
Tue Dec 03 19:27:38.735 2024 UTC [0x0x16debb000] ERROR (TunNetworkInterface:268) ---                 destinationSubnetMask = 255.255.255.255
Tue Dec 03 19:27:38.735 2024 UTC [0x0x16debb000] ERROR (TunNetworkInterface:268) ---             },
Tue Dec 03 19:27:38.735 2024 UTC [0x0x16debb000] ERROR (TunNetworkInterface:268) ---             {
Tue Dec 03 19:27:38.735 2024 UTC [0x0x16debb000] ERROR (TunNetworkInterface:268) ---                 destinationAddress = 104.18.34.132
Tue Dec 03 19:27:38.735 2024 UTC [0x0x16debb000] ERROR (TunNetworkInterface:268) ---                 destinationSubnetMask = 255.255.255.255
Tue Dec 03 19:27:38.735 2024 UTC [0x0x16debb000] ERROR (TunNetworkInterface:268) ---             },
Tue Dec 03 19:27:38.735 2024 UTC [0x0x16debb000] ERROR (TunNetworkInterface:268) ---         )
Tue Dec 03 19:27:38.735 2024 UTC [0x0x16debb000] ERROR (TunNetworkInterface:268) ---         overridePrimary = NO
Tue Dec 03 19:27:38.735 2024 UTC [0x0x16debb000] ERROR (TunNetworkInterface:268) ---     }
Tue Dec 03 19:27:38.735 2024 UTC [0x0x16debb000] ERROR (TunNetworkInterface:268) ---     MTU = 1436
Tue Dec 03 19:27:38.735 2024 UTC [0x0x16debb000] ERROR (TunNetworkInterface:268) --- }
0
sboanr OP
2w

And when connected to the VPN I see the routes setup correctly as well as the DNS settings. But when I try to resolve a host name via dig or nslookup I see no traffic being sent to the utun interface. I have also disabled the firewall and I have no other filters installed.

❯ netstat -nr -f inet
Routing tables

Internet:
Destination        Gateway            Flags               Netif Expire
default            link#24            UCSg                utun9       
default            192.168.0.1        UGScIg                en0       
20.253.190.7       192.168.0.1        UGHS                  en0       
100.127.255.254    link#24            UHWIig              utun9       
104.18.34.132      192.168.0.1        UGHS                  en0       
127                127.0.0.1          UCS                   lo0       
127.0.0.1          127.0.0.1          UH                    lo0       
169.254            link#11            UCS                   en0      !
172.64.153.124     192.168.0.1        UGHS                  en0       
172.86.175.254     172.86.175.254     UH                  utun9       
192.168.0          link#11            UCS                   en0      !
192.168.0.1/32     link#11            UCS                   en0      !
192.168.0.1        2e:30:44:55:b6:eb  UHLWIir               en0   1188
192.168.0.232/32   link#11            UCS                   en0      !
192.168.0.232      8e:7c:9d:b1:c4:8b  UHLWI                 lo0       
192.168.0.255      ff:ff:ff:ff:ff:ff  UHLWbI                en0      !
224.0.0/4          link#24            UmCS                utun9       
224.0.0/4          link#11            UmCSI                 en0      !
224.0.0.251        1:0:5e:0:0:fb      UHmLWI                en0       
224.0.0.251        link#24            UHmW3I              utun9   3575
255.255.255.255/32 link#24            UCS                 utun9       
255.255.255.255/32 link#11            UCSI                  en0      !
❯ cat /etc/resolv.conf
#
# macOS Notice
#
# This file is not consulted for DNS hostname resolution, address
# resolution, or the DNS query routing mechanism used by most
# processes on this system.
#
# To view the DNS configuration used by this system, use:
#   scutil --dns
#
# SEE ALSO
#   dns-sd(1), scutil(8)
#
# This file is automatically generated.
#
nameserver 100.127.255.254
❯ scutil --dns
DNS configuration

resolver #1
  nameserver[0] : 100.127.255.254
  if_index : 24 (utun9)
  flags    : Supplemental, Request A records
  reach    : 0x00000003 (Reachable,Transient Connection)
  order    : 104200

resolver #2
  nameserver[0] : 100.127.255.254
  if_index : 24 (utun9)
  flags    : Request A records
  reach    : 0x00000003 (Reachable,Transient Connection)
  order    : 200000
0
sboanr OP
2w

@DTS Engineer

I found another Intel based Mac to test on and it also did not work so for now let's say that the Intel vs Apple Silicon is a red herring and continue to look at this as a 15.1.1 issue alone.

Scott

0
DTS Engineer OP
Apple
2w
so for now let's say that the Intel vs Apple Silicon is a red herring

*phew*

I’m glad to hear that because I was struggling to think of any way the CPU architecture would be relevant.

I’m gonna role out my standard advice here:

  • Test on a ‘clean’ machine, one that’s never seen your software before. Oh, and if you work for a large enterprise, make sure it doesn’t have your enterprise’s security software installed.

  • And the best way to do that is to use a VM, so you can restore to a clean snapshot between each test.

Good luck! And please keep me apprised of your findings.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

0
sboanr OP
2w

@DTS Engineer

So there is nothing else? The fact that our software worked just fine on 15.0 when it first came out and the ONLY thing that changed is upgrading to 15.1.1 points specifically to an issue on the macOS side. I have tested this on various machines, even a brand new out of the box Macbook Air, and get the same results.

The fact that no packets are even making it to the TUN interface (via wireshark) points to something in the OS and not our client, correct? Especially when it worked on 15.0.

I want to submit a Code Level support ticket but extracting out the code into a focused Xcode project is going to be time consuming.

0
sboanr OP
2w

@DTS Engineer

One last question (I guess) do you see anything in how the PTP is configured above that would cause an issue?

0
sboanr OP
2w
Accepted Answer

@DTS Engineer

Thank you for all your help on this. I am going to close this as answered. In debugging I found that a thread in the extension was actually throwing a BAD_INSTRUCTION error in the OpenSSL lib we were using. After upgrading to a new version of OpenSSL that fixed the BAD_INSTRUCTION everything now works. Of course the mystery is why does the same client work just fine on 14.x but 15.x it did not?
No matter it seems to be working now and that is all that matters. :-) Again thank you.

0
DTS Engineer OP
Apple
2w

Glad to hear you got this sorted.

Of course the mystery is why does the same client work just fine on 14.x but 15.x it did not?

Yeah, that’s tricky. If you really want to know the answer you have to dig into the code that’s crashing. Sometimes it’s something innocuous, like a change in memory allocation patterns, that just happens to trigger a change in the undefined behaviour.

I’m in two minds about this:

  • Sometimes it’s good to just ‘take the win’.

  • OTOH, sometimes bugs that mysteriously disappear will mysteriously reappear.

*shrug*

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

0
VPN network extension not working after 15.1.1 upgrade
First post date Last post date
 
 
Q