How can I connect to a proxy server?

Question

Created Sep ’17

Replies 7

Boosts 1

Views 5.9k

Participants 2

I'm talking about using the proxy settings under "System Preferences" : Network : Advanced : Proxies. You can read the information from there using the System Configuration library. I got some questions on these:

The System Configuration dictionary has the values for the "Exclude Simple Hostnames" and "Bypass proxy settings for these Hosts & Domains", but how do you actually use them to check if a target server should be channeled through a proxy?
The proxy server entry has a hostname and port, and optionally a username & password. I think the System Configuration dictionary only has the hostname & port. How can I tell if there's also a username & password? And where and what those values are? I guess the user's Keychain database is involved.
How do you send the channling request to the proxy server? Isn't the HTTP CONNECT command involved? But how do I put the host, port, username, and password into the URLRequest object?

(I thought there were posts about these things in an old Apple developer mailing list many years ago, but I can't find where.)

Can the proxy-to-NSSessionStreamTask process described towards the end of WWDC 2015 Session 711 ("Networking with NSURLSession") be used with these classic proxies? The presenters had code like:

func URLSession(session: NSURLSession, dataTask: NSURLSessionDataTask, didReceiveResponse response: NSURLResponse, completionHandler: (NSURLSessionResponseDisposition) -> Void)
{
   completionHandler(.BecomeStream)
}

func URLSession(session: NSURLSession, dataTask: NSURLSessionDataTask, didBecomeStreamTask streamTask: NSURLSessionStreamTask)
{
}

I'm just wondering how to create the proxy-request NSURLSessionDataTask in the first place.

(Based off a StackOverflow post I made at "How is a proxy connection made with URLSession?")

Boost

Answer 1

DTS Engineer OP

Apple

Sep ’17

macOS supports multiple types of proxies. Specifically, HTTP proxies work very differently from SOCKS proxies. What sort of proxy are you asking about here?

The general rule of thumb here is that

NSURLSession

takes care of proxies for you (unlike lower-level CFNetwork APIs). Is that not happening in this case?

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

0

Answer 2

CTMacUser OP

Sep ’17

It's definitately not a SOCKS proxy. (I remember at little bit about those from 20 years a while ago.) It's whatever system the FTP/RTSP/Gopher proxies (in the Advanced Network panel) use, which I think are the HTTP type. The automatic proxy support is done at the top level, and I'm working one level below that by attempting a URLProtocol. (I want to practice and provide an example of this stuff.)

0

Answer 3

CTMacUser OP

Sep ’17

OK, these old proxy techniques use the HTTP CONNECT verb. The request line is supposed to be "CONNECT myHost:myPort HTTP/1.1" (or 1.0). The thing is that "myHost:myPort" is not a URL. It is a URI, though. But you have to initiallize a URLRequest with a URL. (I tried with URLComponents and defining just the host and port, and I got "//myHost:myPort". The extra slahses at the start make it useless here, unless you know some form of CONNECT-URI that allows slash prefixes.)

Is there another type of proxy that lets you put full URLs on the CONNECT line? Or doesn't use CONNECT at all? I want to know what kind of proxy connections the authors at Apple (at least the ones at WWDC 2015) had in mind when designing the data-request-to-stream-task conversion delegate method.

If I can't customize URLRequest the way I need, then I have to resort to the CFHTTPMessage API (or straight up manual work).

0

Answer 4

CTMacUser OP

Sep ’17

Just used the playground to see that "myHost:myPort" (with an actual number for "myPort") is accepted by the URL initializer. I don't know how the system interprets it, but as long as it goes unmolested in the HTTP request line, I won't care.

Time for more experimentation....

0

Answer 5

DTS Engineer OP

Apple

Sep ’17

After responding on your other thread I have some more input here.

As far as I can tell Gopher has no specific affordance for proxies. Certainly, there’s no mention of them in RFC 1436. Consequently Apple platforms provide no specific support for Gopher proxies except the ability to set the proxy via the Network preferences panel on macOS. That generates entries in the System Configuration framework preferences and the keychain, but there’s no Gopher-specific stuff there; the Gopher proxy info just follows along with all the other proxy support.

Furthermore, I strongly suspect that Gopher proxy support is only present here because it was a feature of Internet Config back in the day (sacré bleu!, IC has a Wikipedia page).

The upshot of this is that you’ll only be able to support a proxy if that proxy supports general TCP connections. That means either SOCKS or an HTTP[S] proxy via the

CONNECT

method.

For SOCKS you can definitely enable it on CFSocketStream with the code below. This works on both macOS and iOS.

let success = streams.inputStream.setProperty([
    kCFStreamPropertySOCKSProxyHost: "socksproxy.example.com",
    kCFStreamPropertySOCKSProxyPort: 54321,
    kCFStreamPropertySOCKSVersion: kCFStreamSocketSOCKSVersion5
], forKey: Stream.PropertyKey(kCFStreamPropertySOCKSProxy as String))

I had assumed that the following would work with

URLSessionStreamTask

, but I can’t get it to work on either platform.

config.connectionProxyDictionary = [
    "SOCKSEnable": 1,
    "SOCKSProxy": "192.168.1.187",
    "SOCKSPort": 54321,
    kCFStreamPropertySOCKSVersion: kCFStreamSocketSOCKSVersion5,
]

Note I’m using constants strings here because there’s an ongoing issue with cross-platform availability for proxy configuration symbols. See this thread for details.

For an HTTP[S] proxy with the

CONNECT

method, you have a number of options:

You can implement that yourself on top of any TCP API. This isn’t a hugely difficult task given that you can use CFHTTPMessage for HTTP rendering and parsing.
You can use the built-in proxy support in
```
URLSessionStreamTask
```
by configuring the proxies as shown below.
```
config.connectionProxyDictionary = [
    "HTTPSEnable": 1,
    "HTTPSProxy": "192.168.1.187",
    "HTTPSPort": 54321,
]
```
IMPORTANT This doesn’t work on the current public OS releases (iOS 10.3.3 and macOS 10.12.6) but it does work on the current beta OS releases (iOS 11 beta, macOS 10.13 beta).

One thing that won’t work is using a

URLSessionDataTask

for the

CONNECT

request and then switching that to a

URLSessionStreamTask

via

ResponseDisposition.becomeStream

. The issue, as you’ve determined, is that there’s no way to get the URL through to the proxy;

URLSessionDataTask

insists on sending the path component of the URL with a leading

.

Note It turns out that

ResponseDisposition.becomeStream

is less useful than it should be for other reasons as well )-:

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

0

Answer 6

CTMacUser OP

Sep ’17

I was about to post before reloading and seeing your 2017-9-12T2:27 post. I realized that using URLRequest still can't work. Let's take "http://www.example.com/path/to/greatness". The request line will contain "/path/to/greatness", the "Host" header will have "www.example.com:80" and the request will be sent to www.example.com. But for a proxy CONNECT, the request-line and host-header are set to "myDesiredHost:myDesiredPort" while the request has to be sent to 'myProxyServer:itsProxyPort". These differ, but I don't think URLRequest was designed to handle that.

So, if I set the HTTP proxy in the stream-task to "myProxyServer" (starting from High Sierra), I can still point the task to "myDesiredHost"? And the CONNECT (or whatever) will happen behind the scenes?! And the data will go back & forth to "myDesiredHost" (with "myProxyServer" in the middle)? Does that proxy dictionary support username & password too?

Did you file an internal bug about SOCKS support in URLSessionStreamTask? In the following note, on cross-plaform symbol availibility, did you mean to link to my other thread?

0

Answer 7

DTS Engineer OP

Apple

Sep ’17

So, if I set the HTTP proxy in the stream-task to "myProxyServer" (starting from High Sierra), I can still point the task to "myDesiredHost"? And the CONNECT (or whatever) will happen behind the scenes?

That’s what I saw in my testing.

Does that proxy dictionary support username & password too?

No. However, it should:

Resolve proxy authentication challenges automatically via
```
URLCredentialStorage
```
(and thence the keychain)
Pass unresolved authentication challenges to the session delegate

You may encounter problems with the second point, where a failed challenge triggers the default system-wide proxy authentication UI even in situations where you’ve customised the proxies and stand ready to respond to any authentication challenges (r. 21847695).

Did you file an internal bug about SOCKS support in URLSessionStreamTask?

Not personally, but I did run it past CFNetwork Engineering just to make sure I wasn’t missing something obvious and it turns out that this is is a bug (r. 34396519).

URLSessionStreamTask

will use a SOCKS proxy it’s configured system-wide but not if you configure one programmatically via

connectionProxyDictionary

.

In the following note, on cross-plaform symbol availibility, did you mean to link to my other thread?

No, sorry, that’s a bug in my post. I’ve fixed it now.

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

0