User Agent no longer present on CONNECT requests starting from iOS 15

It seems the User Agent is no longer included when iOS 15 sends CONNECT requests to a proxy. Is this the expected behaviour? @eskimo?

Post not yet marked as solved Up vote post of raspu Down vote post of raspu
6.2k views

Answers

I'll let Quinn weigh in here, but just to get some more information on this question, can you describe which API you are using and what type of proxy your client side connection is interacting with?

Matt Eaton
DTS Engineering, CoreOS
meaton3@apple.com

Hi, the proxy is Charles. We use it to validate our App's traffic, we have some rules that try to match the User-Agent. Of course we can achieve this by other means but we would like to know if this is a permanent change and not an issue before thinking in doing so.

  • Oh, and we are using URLSession, all requests seem to be impacted. (@meaton)

Add a Comment

I have the same question actually. I've posted about similar changes in iOS 14 before (https://developer.apple.com/forums/thread/657824). Is there any chance this can get reinstated?

Matt and I discussed this with the team. It’s likely that this change is unexpected fallout from the proxy support unification work we’ve been doing recently [1]. Having said that, we’re disinclined to fix this because the user agent string is a potential source of personal information.

So, Cougnes, if I’m reading your other thread correctly it seems that we ‘broke’ this in iOS 14. Is that right?

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] To unify the proxy implementation between CFNetwork and Network framework.

  • Hey @eskimo, out of curiosity, wouldn’t this change be a break on industry conventions? The HTTP standard recommends on always including a User-Agent…

Add a Comment

No, not exactly. In that case last year, manually set UA’s would not be respected in CONNECT requests for any NSURLRequests.

The behavior that I currently see is that no user agent whatsoever is passed for CONNECT requests. This feels quite arbitrary to me, as all other requests (GET/POST/…) do function as expected.

I can confirm that this bug is still present in iOS15 beta 2.

I can confirm that this bug is still present in iOS15 beta 2.

Did you file a bug about this? ’cause at this point I’m not sure that Apple knows about it officially.

I can’t guarantee that we’ll actually change this behaviour back. Still, if this is important to you then your best option is to file a bug with a clear explanation as to what it’s breaking.

Please post your bug number, just for the record.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

  • I have been following this thread, as in our company we use a proxy in the office network that whitelists traffic based on User-Agent. We are currently trying to find a way to work around this issue. I have filed a bug a week or so ago in the feedback assistant. Bug number: FB9186563.

  • I’m assuming the above mentioned issues result in the same behavior I’m noticing here (no UA’s for CONNECT requests). Would it be useful to file another bug report?

  • Yes, sorry, was not clear maybe. The CONNECT requests are missing the User-Agent, this is the issue that is affecting our setup.

Add a Comment

sven-m wrote:

Bug number: FB9186563.

Thanks.


Cougnes wrote:

Would it be useful to file another bug report?

Duplicate bug reports are most helpful when they contain new diagnostic info (not really relevant here) or new information about the impact of the issues (possibly relevant here).

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

  • I received a response on my bug report.

    [Response] Thanks for contacting us. We have additional information.The User-Agent leaks information about the client, by design. It is not strictly required by the specification. For this reason we have chosen to stop leaking that information. Please close this report. Thank you. To close your feedback, click the Actions button, choose Close Feedback and click Close. [End of response]

    I am instructed to close my report, but I do not agree with that. They mention the word "leak" and "by design". I disagree with this, since the User-Agent is part of a greater ecosystem of interoperability and IMO this design is flawed.

Add a Comment

I have the same behavior while using on Device VPN (NEPacketTunnelProvider). No User Agents (at least 95% of the times). Randomly, I see few user Agents reported on some traffic.

We have a webpage that renders different functionality / content based on the operating system. We user UserAgent to match for OS

@eskimo

  • Did you report it as a bug as well?

  • Bug # FB9329234

  • I received response from Apple for bug# FB9329234, and they claim this is working by design.

    @eskimo - Since this is done as part of Apple's proxy unification, does this only impact iOS 15 and above or it could be applicable to all OS versions?

Add a Comment

Update: this bug still exists in iOS 15 beta 3.

Is there any movement in the bug reports mentioned in this thread?

Is there any movement in the bug reports mentioned in this thread?

I have no info to share here (other than to say that the bugs referenced above have all landed in the right place).

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Just to keep this thread up to date: This behavior is still present on iOS 15 beta 6.

  • Yes, I agree. The behavior is the same with Beta 6. Do we know if there is any traction on this issue?

  • No idea...

Add a Comment

Still the same behavior for iOS 15 beta 8.

This is an interesting issue. Just catching up, but it sounds like this was done inadvertently but that Apple is "disinclined to fix this because the user agent string is a potential source of personal information."? I personally feel that not fixing this inadvertent bug is a mistake and is likely to expose much more sensitive personally identifiable information than anything I've ever seen in the User-Agent header.

There are multiple organizations that either opt to or are required to intercept SSL/TLS requests to inspect traffic for malicious code or inappropriate use. In order to aid with preserving privacy many of these organizations will only match specific user agent headers in the HTTP CONNECT request (i.e. for web browsers) in order to avoid decrypting other potentially sensitive information and/or breaking communication for apps that are using certificate pinning. By removing this header in HTTP CONNECT requests it will mean these organizations will start attempting to decrypt and inspect ALL traffic going through these proxies. This will likely break communication for many apps using certificate pinning and unnecessarily expose potentially sensitive information that the organization (or school, as there are many state laws requiring this type of monitoring) would have preferred to remain private.

Ensuring this header is present (and contains the User-Agent information for the app making the request) will protect sensitive personally identifiable information in addition to ensuring apps utilizing certificate pinning will continue to work unhindered.

Please reconsider your position on this issue.

Thank you!

  • Interesting! I think you should create a bug report if you want them to consider this scenario as well...

  • Agreed. That's the best way to get this feedback in front of the folks making the decisions here.

  • Ok thanks, just submitted.

Add a Comment

just submitted

Thanks. But please post your bug number, just for the record.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Add a Comment