Network Extensions for the Modern Mac

Hello. Welcome to Network Extensions for Modern macOS. My name is Jamie Wood. I'm a Software Engineer working on Internet Technologies at Apple. And I am thrilled to be here today to tell you about a bunch of great new powerful APIs that we've added in macOS Catalina that allow you to create apps that extend and customize the networking capabilities of macOS without the use of Network Kernel Extensions.

To start off, I want to say thank you for your feedback. Over the past few years here at WWDC we've asked you to file bugs and give us feedback about how you're making use of Network Kernel Extensions in your apps today. We got a lot of great feedback. We've taken your feedback, and we came up with a set of categories of apps where on a macOS Mojavi and earlier you really need to use Network Kernel Extensions to fully implement apps in these categories.

So today I want to take you on a journey for each one of these app categories and talk about all the great new APIs we've added in macOS Catalina that allow you to create apps in these categories without using Network Kernel Extensions. So let's go ahead and get started.

First, I want to talk about Content Filter Apps. One example of a Content Filter App is a Personal Firewall App. These are apps that examine the network traffic as it's flowing through the system and block traffic that's deemed to be malicious in some way.

Another example of a Content Filter App is a Parental Controls App. This is an app that focuses on web-browsing activity and blocks access to websites that are deemed inappropriate for children.

Another example of a Content Filter App is an app that doesn't actively block any network traffic but instead just keeps a record of network activity on the Mac so that that log of no activity can be analyzed later, for example, to determine when some sensitive data was transmitted.

So before I talk about the APIs we've added that allow you to create Content Filter Apps, I want to talk about some particular runtime requirements that Content Filter Apps have. So the code in your Content Filter App that is actually filtering network traffic has some specific runtime requirements.

Your code needs to be running all the time, and it needs to be running even when there's no user logged into the system.

For example, in your Parental Controls App, you app needs to be doing its job of blocking access to inappropriate websites even when your app isn't actually running.

In your Personal Firewall App, your app needs to be doing its job of protecting the Mac from incoming attacks coming in over the network even if there's no user logged into the system.

Now, when you've implemented your content filter code inside of a Kernel Extension, these runtime requirements are obviously met because your code is running in the Kernel. So it's running all the time and it's running even when there's no user logged into the system. So to satisfy these runtime requirements in user space, we've introduced a new technology in macOS Catalina called System Extensions.

Now you're probably familiar with app extensions. These are bundles of executable code that you can use on macOS to extend and customize various aspects of the macOS user experience.

So system extensions share a lot of similarities with app extensions. Like app extensions, system extensions are packaged inside of your app and they're completely managed by the operating system. So this is great because it means you don't need to write any customer installer package to place your system extensions somewhere in a file system, and you don't need to write an uninstaller to remove your system extension when the user uninstalls your app. Also you don't need to worry about starting and stopping your system extension. The operating system will run your system extension as needed. Another similarity with app extensions and system extensions is that system extensions are very easy to develop and debug. You can use all the regular tools you use to develop any regular app -- Xcode, LLDB, Instruments.

This is in contrast to Kernel Extensions which are notoriously difficult to develop and debug. You frequently have to reboot as you're developing your extension. And to debug a Kernel Extension you have to have two separate machines and if you do manage to connect these two machines together and drop into the debugger in your Kernel Extension Code, single-stepping through your source code is a very dicey proposition and -- if it works at all. Unlike app extensions, system extensions run independently of any user logged into the system. So systems extensions are really an ideal place for you to be running your network processing code like your content filter code.

For information and details about system extensions and for some other use cases of system extensions, please see the session that happened earlier this week, System Extensions and Driver Kit.

All right. So you use system extensions to implement several different apps in these categories that I've listed here -- Content Filter Apps, Transparent Proxy Apps, DNS Proxy Apps and VPN Apps. Now I want to dive into the APIs we've added that allow you to implement Content Filter Apps.

So the Content Filter APIs are in the Network Extension Framework and these APIs were first introduced back in iOS 9. And so what we've done in macOS Catalina is brought these APIs over and made them available on the Mac and added a bunch of great new enhancements that make these APIs even better. So let's take a look at the Content Filter APIs and how you use them in your app. So in your main UI App, you use NEFilterManager to create a content filter configuration. The content filter configuration registers your content filter with the system so the system knows how to run your filter. You also create a system extension. This is where your code that actually filters network content will run.

The Content Filter APIs allow you to filter network content at two different layers. You can filter content at the flow layer or at the packet layer.

For flow layer filtering, you create a subclass of any data filter provider.

Once your content filter configuration is registered with the system, and your filter's up and running, the system, as new connections, as new TCP and UDP flows of network data are created on the system, those flows get passed to your -- NEFilterDataProvider subclass represented as individual NEFilterFlowObjects. It's then the responsibility of your subclass to make a allow or drop decision about each individual flow.

You can make this decision about each flow at any point in the lifetime of the flow. You can make it right up front when the flow is first opened or you can wait after you've seen some amount of the flow's data.

I want to note here that the NEFilterDataProvider class gives you read-only access to flows.

You can't modify any aspect of a flow, including any of the flow's data.

By default, the system will pass every single flow of TCP and UDP data to your NEFilterDataProvider subclass. If this isn't exactly what you want -- for example, if you're writing a Parental Controls App, so you're only interested in Web traffic, you use NEFilter settings to create a set of rules that inform the system about the flows that you want to see in your filter. So that's how flow level filtering works.

If you want to filter traffic at the packet layer, you create a subclass of NEFilterPacketProvider in your system extension and as network packets are flowing through the system, the system will pass those packets to your FilterPacketProvider subclass as individual packet objects, and you make a Allow or Drop decision about each individual packet. Okay. So there's a brief overview of the Content Filter APIs and how you use them in your app. Next I want to give you a brief demonstration of an app that uses the system extensions and content filter APIs to implement a firewall.

So the functionality of my app is very simple. I'm going to prompt the user to allow or deny incoming TCP connections on Port 8888. So let me go ahead and run the app and show you how it works. So the app's called Simple Firewall. Go ahead and run the app.

And you can see my UI indicator here. The red dot is showing that my content filter is not currently running. So I'm going to go ahead and click start.

Alright. So I get this dialog from the system indicating that my system extension has been blocked from running. Now system extensions are very powerful. They give you the ability to do lots of things on the system including looking at network traffic that's flowing through the system. So we want to make sure that we get the user's permission before allowing system extensions to run. So I'm going to go ahead and open Security Preferences.

This will take me to the Security and Privacy Preferences pane. I'll provide my admin credentials and go ahead and click allow to allow my system extension to run.

The network extension framework also prompts the user to confirm that they want to allow the system extension to filter network traffic on the Mac. So go ahead and click allow. All right. So, now back in simple firewall we can see that my content filter is now running. So let's go ahead and connect to Port 8888 here on my local Mac and see what happens. So I'm running a webserver on Port 8888. So I'm going to go ahead bring up Safari. And I have it bookmarked to my local webserver. So I'll click on that. So the webpage starts to load, but you can see it pauses here. And sure enough, over in the simple firewall app I have this dialog inform me that a new connection has been created on Port 8888 asking me if I want to allow or deny. So I'll click allow, and the webpage loads.

So, cool. My app is working.

Now let's go ahead and take a look at some of the code in simple firewall to see how it makes use of the system extensions and content filter APIs.

So over here you can see my project. I have two different targets. I have the SimpleFirewall Target which is my main UI App.

And I have the SimpleFirewallExtension Target which is my System Extension.

Let's start off my looking at some of the code in the app. We'll take a look at the implementation of my main View Controller Class.

And I want to start by looking at the startFilter function.

So this is the function that was called when I clicked on the start button in the SimpleFirewall UI. I start off by getting the bundle identifier of my system extension and using that to create a system extension activationRequest. I set the view controller object as the delegate of the activationRequest so that it will be notified when the Request is complete. Once the activationRequest is created, I submit it to the OSSystemExtensionManager. This kicks off the process of activating the system extension including prompting the user to allow the SystemExtension to run if necessary.

Okay. So once the user has allowed the SystemExtension to run, my view controller's request did finish with result function gets called.

I make sure that the activation request was completed and I go ahead and create my content filter configuration. So here's where I'm using AnyFilterManager to create my filter configuration and register it with the system. So here you can see I'm setting up some details about my -- on my configuration.

I specify True for filter sockets. This indicates that I'm going to be filtering network traffic at the flow layer.

I set filter packets to False to indicate that I'm not going to be filtering network traffic at the packet layer. I go ahead and Enable my Content Filter Configuration and then register the Configuration with the system by calling Save to Preferences.

So since my Content Filter Configuration is enabled, this will cause the system to start the SystemExtension and start my Content Filter. So let's go ahead and take a look at the implementation of my NEFilterDataProvider subclass running inside of the system extension. So here's by subclass. It's called FilterDataProvider. And I've overridden three different methods in this class -- StartFilter, StopFilter, and HandleNewFlow.

So, first, let's take a look at StartFilter. This is the function that is called when the system starts my content filter. Now, by default, the system is going to pass every single TCP and UDP flow to my content filter. And this isn't really what I want to do here. I'm only interested in inbound TCP connections connecting to Port 8888 on my Mac. So I'm going to create an NEFilterSettingsObject to inform the system about what traffic I want to see.

Now I don't care where the TCP connections are coming from, and I also don't care what address the TCP connections are connecting to on my Mac. So I'm going to create two NEFilter Rules, one with the wildcard IPV4 address and another with the wildcard IPV6 address.

For each Filter Rule that I create, I create an NENetworkRuleObject that specifies the characteristics of the flows that I want to see, that I want the Filter Rule to match.

So for remote network and remote prefix, I'm passing nil and zero. So this means that my filter rule is going to match traffic coming from anywhere. I don't care where it's coming from.

For a local network, I pass in a NWHostEndPoint that I've created using the wildcard address and a local port of 8888. Okay.

So this means that my filter rule's going to match flows that are coming in and connecting to Port 8888 on any address. I specify a protocol of TCP and a direction of Inbound.

I go ahead and create the NEFilterRuleObject, passing in the NENetworkRule, and an action of filter data.

So when a new flow of network data that matches my NENetworkRule is created on the system, the system will pass that flow to my content filter per the filter data action. All right.

So once I've created these NEFilterRules, I go ahead and create my NEFilterSettingsObject, passing in the rules and specifying a default action of Allow. So what this means is if a new flow is created on the system, and it doesn't match any of my filter rules, I want the system to just allow that flow. Don't pass it to my content filter.

I go ahead and call Apply to apply my filter settings to the system and then, when that's complete, I call the StartFilterCompletionHandler to indicate to the system that my filter is now up and running and is ready to start handling network flows. Now let's look at the HandleNewFlow function. So this is the function that's called when a new flow is created that matches my filter rules.

The function takes a parameter, the NEFilterFlowObject that represents the flow, and it returns a new flow verdict to indicate to the system what to do with the flow.

So what I'm doing here is packaging up some details about the flow in a dictionary, and sending that dictionary off to my UI app to prompt the user to allow or deny the flow. Now, getting the user's decision is obviously a very asynchronous process. So while I'm waiting for them to make a decision and go ahead and return a verdict of Pause to the system. So this tells the OS, just hang onto this flow. Don't do anything further with it until I resume the flow.

Once the user makes their decision, I create a new flow verdict of either Allow or Drop, depending upon what the user's decision was, and then I call Resume Flow with the new verdict. Alright. So that was an example of an app that uses the System Extensions and Content Filter APIs to implement a simple firewall.

Next I want to talk about Transparent Proxy Apps. So one example of a Transparent Proxy App is a cloud security app. These are apps that divert traffic destined for specific websites to a cloud service. And that cloud service applies some additional security checks to the traffic such as additional user authentication or authorization.

Another example of a Transparent Proxy App is an app that applies some special transformation to traffic such as applying an encryption algorithm to network traffic or caching resources downloaded over the Web in some special way.

Transparent Proxy Apps can also multiplex multiple flows of network traffic over a single connection.

Or they can use some custom special protocol that reduces network latency. There are a lot of really interesting use cases for Transparent Proxy Apps. So I'm really excited to tell you that in macOS Catalina we've introduced some new APIs in the network extension framework that allow you to create Transparent Proxy Apps without using Kernel Extensions.

So let's go ahead and take a look at these APIs. They're in the NetworkExtension Framework. Let's see how you use them in your app.

So in your main UI App, you use any Transparent Proxy Manager to create Transparent Proxy configurations and register your Transparent Proxy with the system. So your system knows how to run your Transparent Proxy. You also create a system extension. This is where your proxy will run.

So these APIs allow you to proxy flows of network data at the flow layer.

To do this, you create a subclass of NEAppProxyProvider. Now, unlike content filter, by default the system does not divert any flows to your proxy. So you must create a set of NENetworkRules that specify what flows you want to proxy.

So once your Transparent Proxy is up and running and you've installed your NENetworkRules, as new TCP and UDP flows are opened that match your rules, those flows are diverted to your NEAppProxyProvider subclass. From there, it's up to you to completely handle each individual flow.

You can multiplex the flow over another connection, apply your special transformation, whatever you need to do. It's completely up to you. So there's a brief overview of how to use these Transparent Proxy APIs in your app.

Next, let's take a look at DNS Proxy Apps.

Now the DNS protocol is a great protocol, very powerful and useful. But it's not very secure.

So it's pretty easy to spoof DNS responses and cause browsers to go to malicious websites or to spy on somebody's Internet browsing activity simply by looking at the DNS queries that they're sending. So to address these deficiencies, DNS Proxies apply additional security to the DNS Protocol.

For example, the app may apply some encryption to DNS traffic or a proxy DNS traffic over some sort of secure channel.

So I'm pleased to tell you that in macOS Catalina we've introduced some great new APIs that allow you to implement DNS Proxy Apps without using Network Kernel Extensions.

So these APIs are in the NetworkExtension Framework. They were actually introduced in iOS 11, so in macOS Catalina we brought them over and made them available on the Mac. Let's take a look at these APIs and how they work in your app. So in your main new IA app, you'll use NEDNSProxyManager to create your DNS Proxy configuration, register your configuration with the system so the system knows how to run your DNS Proxy. You create a System Extension. This is where your DNS Proxy will run.

And you implement your proxy as a subclass of the NEDNSProxyProvider Class.

So once your DNS Proxy configuration is registered with the system, your system extension is running, the system will start diverting all DNS queries to your NEDNSProxyProvider Subclass. From there it's totally up to you to completely handle each DNS query. You can encrypt it. You can send it over some sort of secure channel. It's totally up to you. Alright. So that's an overview of the DNX Proxy APIs. Next I'm going to talk about VPN Apps. So the classic use case for VPN Apps is to allow companies to provide their employees with secure remote access to their internal corporate network.

Another use case that has grown in popularity a lot in recent years are personal VPN Apps. So these are apps that are used to securely and anonymously browse the Internet. So we actually introduced VPN APIs on macOS back in macOS 10.10. So in this release we've enhanced those APIs to make them even better. Let's take a look at the VPN APIs and how you use them in your app.

So in your main UI app, you use NETunnelProviderManager to create VPN configurations and register your VPN client with the system. You also create a System Extension, which is where your VPN client code will run. You implement your VPN client as a subclass of the NEPacketTunnelProvider class.

The system creates a utun interface corresponding to your NEPacketTunnelProvider.

Your NEPacketTunnelProvider is responsible for telling the system about which networks you want to be routed through your VPN.

So once you've specified your routing rules for your VPN and those are installed in the system, as IP packets get routed to your utun interface per those routes, those packets get diverted to your NEPacketTunnelProvider where you can send those packets through your tunnel connection using your custom tunneling protocol. Okay. So there's a brief overview of how the VPN APIs work. Next I want to talk about a couple of enhancements we've made to the VPN APIs.

So the first is IncludeAllNetworks. This is a new flag you can set on your VPN configuration. This is particularly useful in personal VPN apps. In these apps, it's really important that no traffic leak outside of the VPN tunnel. You want all your traffic to be going through the VPN . Yes.

So by enabling IncludeAllNetworks on your configuration, you cause this to happen. The system will route all traffic through the VPN and if the VPN is not available temporarily for some reason -- for example, if the Mac is switching between the WiFi networks it's connected to or if your VPN is just down for -- temporarily for whatever reason, in those scenarios, traffic will actually be dropped instead of being routed outside of the VPN. Now, if you've enabled IncludeAllNetworks, but you still want to allow access to local network resources such as printers, you can enable ExcludeLocalNetworks to allow that access to still happen. Okay.

So we've also made some enhancements to Per-App VPN.

We've added three new lists of domains that you can use to route traffic to your Per-App VPN. So the way these work is, for each one of these lists, if the corresponding app creates a connection to a host and that host domain matches one of the domains in the list, that connection's traffic will be routed through the Per-App VPN. Let's look at an example. So, if you were using the Mail App, and you have the Mail App set up with two accounts -- you have your personal e-mail account and you have your corporate e-mail account. By specifying the domain of the corporate e-mail server in the mail domain's array, when Mail opens up a new connection to your corporate e-mail server, that connection will be routed through the Per-App VPN, while connections to your personal e-mail server will not be routed through the Per-App VPN.

So the CalendarDomains and ContactsDomains lists behave the same way except for the Calendar App and the Contacts App. Okay.

So that was a brief overview of the VPN APIs that are available on macOS, and some enhancements we've made that allow you to create VPN Apps without the use of Network Kernel Extensions. Next, I want to talk about Virtual Machine Apps. So these are apps that create and manage virtual machines. And, honestly, a virtual machine probably is not very useful if it can't connect to the network.

So on macOS we have the vmnet.framework that allows you to do just that, connect virtual machines to the network. The vmnet.framework was introduced on macOS back in macOS 10.10. But we made a lot of enhancements in this release to give you more ways to connect virtual machines to the network. The way the framework works is it gives you several different modes of connecting virtual machines to the network.

We've made some enhancements to Shared Mode. You can now use IPv6 in Shared Mode. You can specify the range of IPs you want to assign to your virtual machines. And you can set up Port Forwarding Rules between your virtual machines and the network. We've also added a brand new mode called Bridged Mode. In this mode your virtual machines show up on the local network as if they were physically connected to the local network. Okay.

So that's a brief overview of the Virtual Machine APIs you can use to connect virtual machines to the network.

Next, I want to briefly talk about apps that use custom low-layer protocols.

So one example of such an app is an app that needs to communicate with a piece of hardware such as a camera or an audio device, and that device only understands some low-layer protocol like a custom link-layer protocol or a custom IP protocol.

Another example of an app that uses a custom IP protocol, for example, is an app that needs to communicate with other machines on a local network using some highly-optimized protocol.

So I'm pleased to announce that in macOS Catalina we've introduced some new APIs that allow you to communicate over the network using custom low-layer protocols without the use of a Kernel Extension.

First, let's look at the API for Custom IP Protocols. This is a new API in a Network Framework.

The way this works is in your app you create a new kind of NWParameters object specifying the identifier number for your custom IP protocol. You then use that NWParameters object to create an NWConnection. And you then use that NWConnection just as you would a TCP or UDP NWConnection to communicate over the network using your Custom IP Protocol.

For a lot more details about NWConnection, please see last year's talk, "Introducing Network Framework." Now let's look at a brief code sample showing how to use this Custom IP Protocol API.

So, first what I'm doing here is creating an NWParameters object using this new constructor that takes in the identifier number for a Custom IP Protocol. Now, it's important to note here that you must pass the identifier number for a Custom Protocol here. You can't pass the number of a protocol that the system is already handling such as TCP, UDP, or ICMP.

Next, I create the destination that I want to communicate with and I create the NWConnection, passing in the destination and my parameters.

So, from there I use the connection just as I would any other NWConnection, start the connection, and I can start sending/receiving packets using my Custom IP Protocol.

Next, let's look at the Custom Link Layer Protocol APIs. These have also been added to Network Framework.

The way this works is in your app you create a NWEthernetChannelObject specifying the -- your custom ether type that you're going to be using. You then use your Channel Object to communicate over an Ethernet Interface using your custom ether type. Let's look at some code to see how this works. So first I get a reference to the current wired Ethernet Interface.

Then I create my NWEthernetChannel object passing in the Interface and my custom etherType.

Now, just like with the custom IP Protocol API, you must pass a custom etherType here. You can't pass an etherType that the system is already handling such as IP or IPV6.

After creating the channel, I set some callback blocks on the channel. The stateUpdateHandler block gets called as the state of the channel changes. When the channel becomes ready, I can go ahead and start sending and receiving packets that use my custom etherType. The receiveHandler block gets called when a new packet that uses my custom etherType is received from the network.

So after my channel is all set up, I go ahead and start it so I can start communicating using my custom etherType. Great.

So that was a brief overview of the new APIs we've added that allow you to communicate over the network using custom low-layer protocols without the use of a Kernel Extension.

Alright. So we've covered a lot of ground here today, a lot of great new APIs that we've added in macOS Catalina that allow you to create apps in all these categories without the use of Network Kernel Extensions.

So now I want to talk briefly about the future of Network Kernel Extensions.

So Network Kernel Extensions have several problems.

First, they're hard to develop, so I mentioned before if you're testing out new functionality, you probably have to reboot a lot. And also, in the case of Network Kernel Extensions you frequently have to work with some very low-level concepts like doing manual M-Buff Chain Manipulation which is very tricky code. It's very easy to get it wrong.

Also Kernel Extensions are hard to debug. You have to have two separate machines and, as I mentioned before, single stepping through your code can be very tricky if it works at all. Also Kernel Extensions can -- a stability problem in a Kernel Extension can be really catastrophic for the system. So if your Kernel Extension crashes, it just doesn't bring down your app. The entire system reboots, which is extremely disruptive to the user and can lead to serious data loss. So because of all these problems with Kernel Extensions and because we've reached this major milestone on macOS where now we have all these APIs that you can use to create apps without the use of Network Kernel Extensions; in macOS Catalina, we are officially deprecating Network Kernel Extensions.

Now, your existing Network Kernel Extensions should continue to work just fine in macOS Catalina. However, we strongly urge you to please check out all these great new APIs that we've added and start adopting them in your apps, replacing your use of Network Kernel Extensions. It's important that you do this as soon as you can because before too much longer, we will remove support for Network Kernel Extensions entirely for macOS.

Okay. So today we talked about a bunch of great new powerful APIs that we have added in macOS Catalina that allow you to create adapts that filter network content, proxy network content, tunnel network content, connect virtual machines to networks, and communicate over the network using custom low-layer protocols, all without the use of Network Kernel Extensions.

This is great news because we strongly urge you to please adopt these new APIs in your Apps because Network Kernel Extensions are now deprecated and support for them will be removed in a future release.

For more information, please check out the webpage for this session. There you can find a link to the sample simple tunnel code that I demoed here today. We also have a Networking Lab that's actually going on right now. And so we'd love to see you there to answer any questions you may have. Thank you so much for coming. Enjoy the rest of your day. [ Applause ]