DNS Proxy system extension – OSSystemExtensionErrorDomain error 9 “validationFailed” on clean macOS machine

Question

Leo_Nagano OP

Created 5h

Replies 0

Boosts 0

Participants 1

Hi,

I’m implementing a macOS DNS Proxy as a system extension and running into a persistent activation error:

OSSystemExtensionErrorDomain error 9 (validationFailed) with the message: extension category returned error

This happens both on an MDM‑managed Mac and on a completely clean Mac (no MDM, fresh install).

Setup macOS: 15.x (clean machine, no MDM)

Xcode: 16.x

Team ID: AAAAAAA111 (test)

Host app bundle ID: com.example.agent.NetShieldProxy

DNS Proxy system extension bundle ID: com.example.agent.NetShieldProxy.dnsProxy

The DNS Proxy is implemented as a NetworkExtension system extension, not an app extension.

Host app entitlements From codesign -d --entitlements :- /Applications/NetShieldProxy.app:

xml <dict> <key>com.apple.application-identifier</key> <string>AAAAAAA111.com.example.agent.NetShieldProxy</string>

<key>com.apple.developer.system-extension.install</key>
<true/>

<key>com.apple.developer.team-identifier</key>
<string>AAAAAAA111</string>

<key>com.apple.security.app-sandbox</key>
<true/>

<key>com.apple.security.application-groups</key>
<array>
    <string>group.com.example.NetShieldmac</string>
</array>

<key>com.apple.security.files.user-selected.read-only</key>
<true/>

</dict> DNS Proxy system extension entitlements From codesign -d --entitlements :- "/Applications/NetShieldProxy.app/Contents/Library/SystemExtensions/com.example.agent.NetShieldProxy.dnsProxy.systemextension/Contents/MacOS/com.example.agent.NetShieldProxy.dnsProxy":

xml <dict> <key>com.apple.application-identifier</key> <string>AAAAAAA111.com.example.agent.NetShieldProxy.dnsProxy</string>

<key>com.apple.developer.networking.networkextension</key>
<array>
    <string>dns-proxy-systemextension</string>
</array>

<key>com.apple.developer.team-identifier</key>
<string>AAAAAAA111</string>

<key>com.apple.security.application-groups</key>
<array>
    <string>group.com.example.NetShieldmac</string>
    <string>group.example.NetShieldmac</string>
    <string>group.example.agent.enterprise.macos</string>
    <string>group.example.com.NetShieldmac</string>
</array>

</dict> Both host and extension binaries are signed with the same Developer ID Application certificate for Team AAAAAAA111, and codesign -dv --verbose=4 shows matching TeamIdentifier and Authority for both.

DNS Proxy system extension Info.plist On the clean Mac, from:

bash plutil -p "/Applications/NetShieldProxy.app/Contents/Library/SystemExtensions/com.example.agent.NetShieldProxy.dnsProxy.systemextension/Contents/Info.plist" I get:

json { "CFBundleExecutable" => "com.example.agent.NetShieldProxy.dnsProxy", "CFBundleIdentifier" => "com.example.agent.NetShieldProxy.dnsProxy", "CFBundleName" => "com.example.agent.NetShieldProxy.dnsProxy", "CFBundlePackageType" => "SYSX", "CFBundleShortVersionString" => "1.0.1.8", "CFBundleSupportedPlatforms" => [ "MacOSX" ], "CFBundleVersion" => "0.1.1", "LSMinimumSystemVersion" => "13.5", "NSExtension" => { "NSExtensionPointIdentifier" => "com.apple.dns-proxy", "NSExtensionPrincipalClass" => "com_example_agent_NetShieldProxy_dnsProxy.DNSProxyProvider" }, "NSSystemExtensionUsageDescription" => "SYSTEM_EXTENSION_USAGE_DESCRIPTION" } The DNSProxyProvider class inherits from NEDNSProxyProvider and is built in the system extension target.

Activation code In the host app, I use:

swift import SystemExtensions

final class SystemExtensionActivator: NSObject, OSSystemExtensionRequestDelegate { private let extensionIdentifier = "com.example.agent.NetShieldProxy.dnsProxy"

func activate(completion: @escaping (Bool) -> Void) {
    let request = OSSystemExtensionRequest.activationRequest(
        forExtensionWithIdentifier: extensionIdentifier,
        queue: .main
    )
    request.delegate = self
    OSSystemExtensionManager.shared.submitRequest(request)
}

func request(_ request: OSSystemExtensionRequest,
             didFailWithError error: Error) {
    let nsError = error as NSError
    print("Activation failed:", nsError)
}

func request(_ request: OSSystemExtensionRequest,
             didFinishWithResult result: OSSystemExtensionRequest.Result) {
    print("Result:", result.rawValue)
}

} Runtime behavior on a clean Mac (no MDM) config.plist is created under /Library/Application Support/NetShield (via a root shell script).

A daemon runs, contacts our backend, and writes /Library/Application Support/NetShield/state.plist with a valid dnsToken and other fields.

The app NetShieldProxy.app is installed via a notarized, stapled Developer ID .pkg.

The extension bundle is present at: /Applications/NetShieldProxy.app/Contents/Library/SystemExtensions/com.example.agent.NetShieldProxy.dnsProxy.systemextension.

When I press Activate DNS Proxy in the UI, I see in the unified log:

text NetShieldProxy: [com.example.agent:SystemExtensionActivator] Requesting activation for system extension: com.example.agent.NetShieldProxy.dnsProxy NetShieldProxy: [com.example.agent:SystemExtensionActivator] SystemExtensionActivator - activation failed: extension category returned error (domain=OSSystemExtensionErrorDomain code=9) NetShieldProxy: [com.example.agent:SystemExtensionActivator] SystemExtensionActivator - OSSystemExtensionError code enum: 9 NetShieldProxy: [com.example.agent:SystemExtensionActivator] SystemExtensionActivator - validationFailed And:

bash systemextensionsctl list

-> 0 extension(s)

There is no prompt in Privacy & Security on this clean Mac.

Question Given:

The extension is packaged as a system extension (CFBundlePackageType = SYSX) with NSExtensionPointIdentifier = "com.apple.dns-proxy".

Host and extension share the same Team ID and Developer ID Application cert.

Entitlements on the target machine match the provisioning profile and Apple’s docs for DNS Proxy system extensions (dns-proxy-systemextension).

This is happening on a clean Mac with no MDM profiles at all.

What are the likely reasons for OSSystemExtensionErrorDomain error 9 (validationFailed) with "extension category returned error" in this DNS Proxy system extension scenario?

Is there any additional configuration required for DNS Proxy system extensions (beyond entitlements and Info.plist) that could trigger this category-level validation failure?

Any guidance or examples of a working DNS Proxy system extension configuration (host entitlements + extension Info.plist + entitlements) would be greatly appreciated.

Thanks!

Boost