Hi all,
I have a working macOS (Intel) system extension app that currently uses only a Content Filter (NEFilterDataProvider). I need to capture/log HTTP and HTTPS traffic in plain text, and I understand NETransparentProxyProvider is the right extension type for that.
For HTTPS I will need TLS inspection / a MITM proxy — I’m new to that and unsure how complex it will be.
For DNS data (in plain text), can I use the same extension, or do I need a separate extension type such as NEPacketTunnelProvider, NEFilterPacketProvider, or NEDNSProxyProvider?
Current architecture:
Two Xcode targets: MainApp and a SystemExtension target.
The SystemExtension target contains multiple network extension types.
MainApp ↔ SystemExtension communicate via a bidirectional NSXPC connection.
I can already enable two extensions (Content Filter and TransparentProxy). With the NETransparentProxy, I still need to implement HTTPS capture.
Questions I’d appreciate help with:
Can NETransparentProxy capture the DNS fields I need (dns_hostname, dns_query_type, dns_response_code, dns_answer_number, etc.), or do I need an additional extension type to capture DNS in plain text?
If a separate extension is required, is it possible or problematic to include that extension type (Packet Tunnel / DNS Proxy / etc.) in the same SystemExtension Xcode target as the TransparentProxy?
Any recommended resources or guidance on TLS inspection / MITM proxy setup for capturing HTTPS logs?
There are multiple DNS transport types — am I correct that capturing DNS over UDP (port 53) is not necessarily sufficient? Which DNS types should I plan to handle?
I’ve read that TransparentProxy and other extension types (e.g., Packet Tunnel) cannot coexist in the same Xcode target. Is that true?
Best approach for delivering logs from multiple extensions to the main app (is it feasible)? Or what’s the best way to capture logs so an external/independent process (or C/C++ daemon) can consume them?
Required data to capture (not limited to):
All HTTP/HTTPS (request, body, URL, response, etc.)
DNS fields: dns_hostname, dns_query_type, dns_response_code, dns_answer_number, and other DNS data — all in plain text.
I’ve read various resources but remain unclear which extension(s) to use and whether multiple extension types can be combined in one Xcode target. Please ask if you need more details.
Thank you.
I’m new to that and unsure how complex it will be.
TLS inspection isn’t too hard, but be aware that it won’t work for all clients. Some system services and apps do their own certificate checks — certificate pinning, public key pinning, and so on — and those are incompatible with TLS inspection.
For DNS data … can I use the same extension
You probably can, but IMO it’s better to use a DNS proxy because that’s specifically designed for this task.
You will need to disable opportunistic secure DNS; see this thread.
is it possible … to include that … in the same SystemExtension … ?
You should be able to include your DNS proxy provider in the same sysex. And the fact that you already have this working with content filter and transparent proxy providers suggests that this’ll actually work in practice (-:
am I correct that capturing DNS over UDP (port 53) is not necessarily sufficient?
Yes. You most definitely need to support DNS-over-TCP on port 53.
You also need to disable opportunistic secure DNS, as I mentioned above.
Having said that, this won’t necessarily work for all clients. Some third-party apps have their own DNS resolvers [1] and in such cases there’s no guarantee that you disable security, avoid DNS-over-HTTP, and so on.
I’ve read that TransparentProxy and other extension types … cannot coexist in the same Xcode target.
I’m not aware of any such limitations for sysexen.
It’s certainly true each appex can only contain a single provider type, but that’s not relevant here.
Best approach for delivering logs
IMO you should log using the system log. I explain why, and how, in Your Friend the System Log.
Using the system log complicates the process capturing logs in your main app, but the benefits outweigh the drawbacks IMO. Specifically, it means your log entries are coordinated with NE’s log entries, which is super useful when debugging hard problems.
Of course, others disagree, and you’re free to write or acquire your own logging library.
Finally, if you’re just getting started, I recommend that you review the links in Network Extension Resources.
Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"
[1] We specifically recommend against that, see TN3151, but folks do it anyway.