Use ProxyProvider to sniff packets without using proxy server?

I guess I’m still a bit confused as to your overall goạl. So, some questions:

• Do you want to see packets or flows? That’s a pretty important distinction.

• It sounds like you want to see the traffic and then pass it through to the original server. Is that correct?

• Is this being deployed to customers? Or just some sort of internal test tool?

• Where do you stand with requiring MDM? What about supervised devices?

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Do you want to see packets or flows? That’s a pretty important distinction.

I want to be able to read the outgoing packets (network packets -> [Data]) and save it as a pcap file format. I don't know if this is called packets or packetFlow, perhaps you could explain the difference or link to a correct explanation.

It sounds like you want to see the traffic and then pass it through to the original server. Is that correct?

Yes, exactly. So I don't want to redirect the traffic to a proxy or over a vpn network. Just sniff packets with the NE. Is this even possible?

Is this being deployed to customers? Or just some sort of internal test tool?

No, this is just for research purpose and will not be released to the appstore. It should make it easier to get the network traffic of a device, otherwise you have to set up a proxy server in the network, which sniffs all packets like a MITM attack, which is a big overhead to do.

Where do you stand with requiring MDM? What about supervised devices?

Do I need to set up a MDM to use the AppProxyProvider? I thought only for the global proxy I need to use MDM?The same thing with supervised devices, which just give the MDM Admin more control. The problem is that you can only set up a global proxy, which does not fit my g-o-a-l, which is getting the traffic without any external proxy server.

I hope it's clearer what I want to achieve, if not ask again !

I don't know if this is called packets or packet flow, perhaps you could explain the difference or link to a correct explanation.

I can understand your confusion. In standard TCP parlance packets would mean separate IP datagrams whereas a flow would mean the contents of a TCP stream (that is, the user data that’s transported over the TCP connection). Some tools (like

tcpflow

NE uses the term flow in both ways:

• A packet tunnel provider reads and writes packets via NEPacketTunnelFlow

• An app proxy provider reads and writes TCP flows via NEAppProxyTCPFlow, but it can also use NEAppProxyUDPFlow for UDP packets

• A content filter reads flows via NEFilterFlow

If you’re aiming to create a

.pcap

I don't want to redirect the traffic to a proxy or over a vpn network. Just sniff packets with the NE. Is this even possible?

No. Well, you might make some progress but the Network Extension provider architecture was definitely not designed to support this. Specifically:

• Packet tunnel providers have the routing problem I described in your previous thread.

• App proxy providers and content filters don’t see packets, they see flows.

No, this is just for research purpose and will not be released to the appstore. It should make it easier to get the network traffic of a device, otherwise you have to set up a proxy server in the network, which sniffs all packets like a MITM attack, which is a big overhead to do.

Have you looked at RVI? Personally I use this for most packet sniffing tasks, except where I need to see inside HTTPS, in which case I use

mitmproxy

There is also CFNetwork diagnostic logging, which is the place I generally start when debugging networking problems. See QA1887 for the details.

Do I need to set up a MDM to use the AppProxyProvider?

The general rules are:

• Normal VPN (a packet tunnel provider in destination IP routing mode (

  routingMethod

  is

  destinationIP

  ) can be used on any device.

• Per-app VPN (an app proxy provider, or a packet tunnel in source application mode) requires MDM.

• A content filter provider requires a supervised device.

The last two points only apply to deployed apps; there is specific support for apps under development to bypass these restrictions. This is predicated on the

get-task-allow

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Thanks that helped a lot for a better understanding!

You mentioned the routing problem with the packet tunnel provider. Can you explain this part more precise, I think I don't get it completely, why there is the problem.

"Normal VPN (a packet tunnel provider in destination IP routing mode (

routingMethod

destinationIP

So this means that the VPN is only used, if the destination IP is in the private Network, otherwise the default route throught eth0 is used?

best regards,

effe2402

So this means that the VPN is only used, if the destination IP is in the private Network, otherwise the default route throught eth0 is used?

A destination IP packet tunnel provider can be set up in two ways:

• Full tunnel — This processes all outbound traffic from the device.

• Split tunnel — This only processes traffic that’s destined for a specific set of networks, like a corporate intranet.

You obviously want to create a full tunnel. You do this by configuring your tunnel with an NEPacketTunnelNetworkSettings object whose

iPv4Settings

includedRoutes

NEIPv4Route.default()

Note In fact it does not see all outgoing traffic. If the traffic matches some other route, it will go via that route. For example:

• Traffic destined to a local network

• Traffic whose source address is bound to a specific interface

This raises the question, how does your packet tunnel provider send encapsulated traffic to the VPN server? If your encapsulated traffic goes via the default route, it’ll come back to your packet tunnel provider and never get anywhere.

It turns out that there are two ways to do this:

• Any addresses you list when you construct the NEIPv4Settings object are automatically excluded.

• You can explicitly exclude addresses via the

  excludedRoutes

  property of the NEIPv4Settings.

Likewise for NEIPv6Settings in both cases.

Your problem is that:

• You want to become the default route

• If you become the default route, you can’t forward traffic to arbitrary addresses

There are various ‘clever’ ways to hack around this but they all have serious drawbacks. The reality is that the Network Extension provider architecture was not designed to support this sort of thing.

Note that I’ve only discussed the outbound path here. The issues with the inbound path are probably worse (-:

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

First of all, how do I reference to you answer(like you do with this grey background) ? 😝

If you become the default route, you can’t forward traffic to arbitrary addresses

And I need to forward the traffic from the vTun over eth0 to the original dest IP, if I get it now.

There are various ‘clever’ ways to hack around this but they all have serious drawbacks.

Can you reference to these hacks or give a short description about, how they are doing a workaround for this problem?

PS: You mentioned above that AppProxyProvider only handles flows instead of packets, which are unused for my pcap-file export purpose, but you also said that appProxy can also read packets with UDPPacketFlows. Are there differences?

Can't I get the packets from the TCPflows?

b r,

effe2402

First of all, how do I reference to you answer (like you do with this grey background) ?

Good question. I can’t see any way to do this using DevForums’s rich text editor. Speaking personally, I don’t use that UI at all. I write my posts in Markdown using a separate text editor. I then have a hot key (yay Keyboard Maestro) that converts the Markdown to HTML, post processes it slightly (yay Python), and pastes it into DevForum’s HTML editor. I’ve found that to be a very efficient workflow.

Can you reference to these hacks or give a short description about, how they are doing a workaround for this problem?

Before I start, I want to stress that I’ve not tried this personally so I don’t know if it’ll work. I also want to be clear that you’re rapidly leaving the design space of Network Extension providers, which means that even if you do get things working today they could well break in the future. Whatever else you do, I encourage you to file an enhancement request requesting an architected solution to this problem. Make sure to post your bug number, just for the record.

Personally, I’d love to see iOS provide a mechanism for on-device packet capture. I think this would be a pretty reasonable fit for the Network Extension provider architecture, but there are lots of other ways you could design this.

In terms of hacks, my idea was:

• Integrate a NAT into your provider, which resolves the incoming traffic issue

• Bind the NAT’s socket to the hardware interface so that its traffic doesn’t get caught by the default route

However, thinking about this further I suspect it won’t be possible because iOS does not give you access to raw IP or libpcap. I’m not sure though. Honestly, the mechanics of implement a NAT are outside of my area of expertise.

PS: You mentioned above that AppProxyProvider only handles flows instead of packets, which are unused for my pcap-file export purpose, but you also said that appProxy can also read packets with UDPPacketFlows. Are there differences?

Yes. A NEAppProxyUDPFlow will give you the UDP packet but you’ll be missing all the headers that you’d expect to see in a PCAP file.

Can't I get the packets from the TCPflows?

No.

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

I did an enhancement request for the on-device packet capture issue with the following bug report number: 31267870.

The mentioned hacks above look very hard to implement, especially I need to handle outbound and inbound traffic different that all packets find the right destination. Wouldn't it simpler to start a on-device/local GCDHTTPProxyServer and connect to it with the TunnelProviderManager like to an external proxy server/vpn server?

Would be there a way to realize this approach? If not, where is the issue(just for better understanding)?

I did an enhancement request for the on-device packet capture issue with the following bug report number: 31267870.

Thanks.

Wouldn't it simpler to start a on-device/local GCDHTTPProxyServer and connect to it with the TunnelProviderManager like to an external proxy server/vpn server?

Probably not )-: Are you proposing to do this as a packet tunnel provider or an app proxy provider? Each has drawbacks:

• Packet tunnel providers only see packets, so you have to reassemble the TCP flows from those packets. This is essentially what a NAT does.

• App proxy providers avoid that issue but:

  • Have serious deployment limitations

  • Don’t let you see the packets, which is a problem given that you want packets in order to create a PCAP file

Also, seeing as you’ve mentioned HTTP here I want to point out the key difference between your situation and a standard debugging HTTP proxy (as per QA1176): a debugging HTTP proxy has user-space coöperation. Imagine the client is fetching the URL

<http://example.com/foo>

GET /foo HTTP/1.1
Host: example.com
… other headers …

OTOH, if it’s connecting to a proxy server

myproxy.local.

GET http://example.com/foo HTTP/1.1
Host: myproxy.local.
… other headers …

If you were to implement a standard HTTP proxy you’d have to find a way to rewrite the former as the latter. This is possible, but not easy. The main difficulty is that it requires you to have access to the TCP flow, which is where I started this response.

It’s also common for proxies like this (so-called transparent proxies) to interfere with the stream in a way that causes compatibility problems.

Note that the above just covers HTTP; HTTPS causes its own special kind of grief (as well it should, see as it’s trying to stop exactly what you’re trying to do here).

And finally, don’t even get me started on HTTP/2 (-:

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Hello , I'm working on the same issue right now ,no idea on how to capture packets while keeping the network working on the right track without proxy server.

Would you please tell me if you have got any further or receive any reply for the bug report?

Hi,

I am also hoping for a clean packet capture implementation wich I think would be a good fit for the NetworkExtensions framework. I belive there are very valid use-cases and applications for this especially in network analysis/monitoring/debugging.

Specifically we would like to use such a mechanism in conjunction with our network monitoring device installed at some other point in the network to perform traffic latency analysis and packet loss analysis.

I have also posted a feature suggestion with bug report number 33695065.

Do you have received any news regarding your enhancemend request?

Best regards,

mw