MacOS: Cannot make requests through proxy advertising "kerberos" or "negotiate" authentication

Hello,


I am facing an issue involving using URLSession to make API requests through an authenticated proxy.


This was originally observed on 10.12.6, but I have also reproduced it on 10.13.3.


I have written a very simple test program to demonstrate:


let url = URL(string: "https://api.myjson.com/bins/vi56v")

let task = URLSession.shared.dataTask(with: url!) {(data, response, error ) in

    guard error == nil else {
        print("returned error")
        return
    }

    guard let content = data else {
        print("No data")
        return
    }

    print("Got data: \(String.init(data: content, encoding: .utf8) ?? "decode failure")")
}

task.resume()

RunLoop.main.run()


Running this without any proxy configured produces the correct result, which lists several Silicon Valley tech companies in JSON form.

Once I configure an autheticated HTTP/HTTPS proxy in System Preferences, however, it stops working. The request fails and the non-specific "Failed to get applicable proxy auth" is printed in the Console. Of interest, the username and password for this proxy are properly configured and present in the keychain, but the app does not seem to even be attemping to read them, because no keychain authorization prompts are seen.


What is of further interest is that this appears to be related to the authentication methods offered by the proxy. The inital 407 rejection contains numerous authentication options (XXXXXXX added to obscure private information):

HTTP/1.1 407 Proxy Authentication Required ( Forefront TMG requires authorization to fulfill the request. Access to the Web Proxy filter is denied.  )
Via: 1.1 XXXXXXX
Proxy-Authenticate: Negotiate
Proxy-Authenticate: Kerberos
Proxy-Authenticate: NTLM
Proxy-Authenticate: Basic realm="XXXXXXXX"
Connection: close
Proxy-Connection: close
Pragma: no-cache
Cache-Control: no-cache
Content-Type: text/html
Content-Length: 712 


However, if I modify the response (using networking tricks) to remove the Kerberos and Negotiate lines, all of a sudden everything starts working great. Keychain access is requested, and I see a proper NTLM 3-way handshake. Since our major requirement is to support NTLM, this is good.


Unfortunately, I think telling our customers "whatever you do, don't enable Kerberos on your proxy" is not going to cut it. I've looked around left and right for some way to override the authentication process for proxies, but all answers (including those from this forum) indicate that "it's magic, there's no way to customize it".


Any ideas on how to proceed?

Do you get the proxy authentication challenges via the session’s delegate callback?

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Thanks for the response! I assume you're referring to the "didReceive challenge" methods of URLSessionDelegate and URLSessionTaskDelegate:

https://developer.apple.com/documentation/foundation/urlsessiondelegate/1409308-urlsession

https://developer.apple.com/documentation/foundation/urlsessiontaskdelegate/1411595-urlsession


If so, no, those methods are not called. As far as I can tell from looking around the internet, it seems this is expected behavior, i.e. MacOS will only call these for regular HTTP authentication, not for proxy authentication, and that instead MacOS is supposed to "magically" look up the proxy credentials.


For the record, here is an expanded test program with the delegates, to ensure I'm not doing something dumb:

class MyDelegate: NSObject, URLSessionDelegate, URLSessionTaskDelegate {

    func urlSession(_ session: URLSession, didReceive challenge: URLAuthenticationChallenge, completionHandler: @escaping (URLSession.AuthChallengeDisposition, URLCredential?) -> Void) {
        print("session challenge")
        completionHandler(.performDefaultHandling, nil)
    }

    func urlSession(_ session: URLSession, task: URLSessionTask, didReceive challenge: URLAuthenticationChallenge, completionHandler: @escaping (URLSession.AuthChallengeDisposition, URLCredential?) -> Void) {
        print("task challenge")
        completionHandler(.performDefaultHandling, nil)
    }
}

let url = URL(string: "https://api.myjson.com/bins/vi56v")
let config = URLSessionConfiguration.default
let delegate = MyDelegate()
let session = URLSession(configuration: config, delegate: delegate, delegateQueue: nil)

let task = session.dataTask(with: url!) {(data, response, error ) in

    guard error == nil else {
        print("returned error")
        return
    }

    guard let content = data else {
        print("No data")
        return
    }

    print("Got data: \(String.init(data: content, encoding: .utf8) ?? "decode failure")")
}

task.resume()

RunLoop.main.run()


I actually get the session-level challenge delegate to fire when NOT using a proxy, but once I add the auth proxy it's back to an immediate error with no delegate being called.

… those methods are not called.

Yeah, that’s pretty much what I expected.

As far as I can tell from looking around the internet, it seems this is expected behavior, i.e. MacOS will only call these for regular HTTP authentication, not for proxy authentication, and that instead MacOS is supposed to "magically" look up the proxy credentials.

Yeah, that depends on who you talk to (-: Seriously though, the original design of the Foundation URL loading system gave apps first crack at all authentication challenges, including those for proxies. Hence the

isProxy
property on
NSURLProtectionSpace
. However, for various reasons that would make me too sad to explain, the currently implementation does not work that way. If a system-wide proxy is configured then the system proxy infrastructure will always attempt to handle the authentication challenge first. And it seems that in your case that infrastructure is not handling things correctly.

We have a bunch of bugs on file about this already [1] but that doesn’t help you right now. For the moment the only answer I can think of is to have the proxy not send out the problematic

Proxy-Authenticate
headers, possibly conditionalising that based on the user agent.

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

[1] Specifically:

  • (r. 19530018) to track this specific problem

  • (r. 30417163) to request a better authentication architecture that makes it harder for problems like this to reoccur in the future

Thanks for the information!


As you can imagine, this is pretty distressing for my company, as we are an enterprise software company, and the largest (and therefore most desirable) customers tend to have complicated network setups which often involve authenticated proxies. This is around 15% of our customer base. So writing an enterprise-grade application with Apple's networking stack is basically impossible as long as these problems exist.


Having any kind of custom behavior on the proxy side is not really possible as customers are deploying standard software built by third party vendors who would not be able to customize for our needs (in the particular case above, the proxy is made by a certain company from Redmond...)


I would really hate to have to rip out our Apple-based networking stack, which is performing admirably in all other cases, and replace it with something else, like cURL. Are there any other workarounds that might be possible that you can find in the associated Radars?


We're also working this issue through our enterprise developer relations person as well, so we will try to get some more info through that channel as well.

Are there any other workarounds that might be possible that you can find in the associated Radars?

No that I can see.

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"
MacOS: Cannot make requests through proxy advertising "kerberos" or "negotiate" authentication
 
 
Q