User Agent no longer present on CONNECT requests starting from iOS 15

I'll let Quinn weigh in here, but just to get some more information on this question, can you describe which API you are using and what type of proxy your client side connection is interacting with?

Matt Eaton
DTS Engineering, CoreOS
meaton3@apple.com

Hi, the proxy is Charles. We use it to validate our App's traffic, we have some rules that try to match the User-Agent. Of course we can achieve this by other means but we would like to know if this is a permanent change and not an issue before thinking in doing so.

I have the same question actually. I've posted about similar changes in iOS 14 before (https://developer.apple.com/forums/thread/657824). Is there any chance this can get reinstated?

Matt and I discussed this with the team. It’s likely that this change is unexpected fallout from the proxy support unification work we’ve been doing recently [1]. Having said that, we’re disinclined to fix this because the user agent string is a potential source of personal information.

So, Cougnes, if I’m reading your other thread correctly it seems that we ‘broke’ this in iOS 14. Is that right?

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] To unify the proxy implementation between CFNetwork and Network framework.

No, not exactly. In that case last year, manually set UA’s would not be respected in CONNECT requests for any NSURLRequests.

The behavior that I currently see is that no user agent whatsoever is passed for CONNECT requests. This feels quite arbitrary to me, as all other requests (GET/POST/…) do function as expected.

I can confirm that this bug is still present in iOS15 beta 2.

I can confirm that this bug is still present in iOS15 beta 2.

Did you file a bug about this? ’cause at this point I’m not sure that Apple knows about it officially.

I can’t guarantee that we’ll actually change this behaviour back. Still, if this is important to you then your best option is to file a bug with a clear explanation as to what it’s breaking.

Please post your bug number, just for the record.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

sven-m wrote:

Bug number: FB9186563.

Thanks.

Cougnes wrote:

Would it be useful to file another bug report?

Duplicate bug reports are most helpful when they contain new diagnostic info (not really relevant here) or new information about the impact of the issues (possibly relevant here).

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

I have the same behavior while using on Device VPN (NEPacketTunnelProvider). No User Agents (at least 95% of the times). Randomly, I see few user Agents reported on some traffic.

We have a webpage that renders different functionality / content based on the operating system. We user UserAgent to match for OS

@eskimo

Update: this bug still exists in iOS 15 beta 3.

Is there any movement in the bug reports mentioned in this thread?

Is there any movement in the bug reports mentioned in this thread?

I have no info to share here (other than to say that the bugs referenced above have all landed in the right place).

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Just to keep this thread up to date: This behavior is still present on iOS 15 beta 6.

Still the same behavior for iOS 15 beta 8.

This is an interesting issue. Just catching up, but it sounds like this was done inadvertently but that Apple is "disinclined to fix this because the user agent string is a potential source of personal information."? I personally feel that not fixing this inadvertent bug is a mistake and is likely to expose much more sensitive personally identifiable information than anything I've ever seen in the User-Agent header.

There are multiple organizations that either opt to or are required to intercept SSL/TLS requests to inspect traffic for malicious code or inappropriate use. In order to aid with preserving privacy many of these organizations will only match specific user agent headers in the HTTP CONNECT request (i.e. for web browsers) in order to avoid decrypting other potentially sensitive information and/or breaking communication for apps that are using certificate pinning. By removing this header in HTTP CONNECT requests it will mean these organizations will start attempting to decrypt and inspect ALL traffic going through these proxies. This will likely break communication for many apps using certificate pinning and unnecessarily expose potentially sensitive information that the organization (or school, as there are many state laws requiring this type of monitoring) would have preferred to remain private.

Ensuring this header is present (and contains the User-Agent information for the app making the request) will protect sensitive personally identifiable information in addition to ensuring apps utilizing certificate pinning will continue to work unhindered.

Please reconsider your position on this issue.

Thank you!

just submitted

Thanks. But please post your bug number, just for the record.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Still the same behavior on iOS 15 GM, any chance this gets fixed on 15.1 @eskimo?

Still the same behavior on iOS 15 [RC]

Correct. I fully expect that it will also work this way on the final released version of iOS 15.

any chance this gets fixed on 15.1

While I can’t predict the future, more’s the pity, I don’t recommend holding your breath.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Quick update: Still an issue in both 15.0.1 and 15.1 Beta 2.

From @rfan1560's comment:

I received a response to my feedback (FB9605687).

Apple Response: Investigation complete - Works as currently designed Please know that our engineering team has determined that this issue behaves as intended based on the information provided. Ppecifying the user-agent is a "should", not a "must".

So @eskimo, is this now considered "by design" then? There's no chance that this will change in future versions of iOS?

This is Symantec's article letting customers know that the User-Agent policies will no longer work, and that a workaround is not available. https://knowledge.broadcom.com/external/article?articleId=223857 In my opinion it's unfortunate (for admins and users alike) that those required to monitor web traffic are now being forced to decrypt (or block) all network traffic originating from iOS 15+ devices due to a change that (from the early posts in this thread) appears to have been unintentional.

appears to have been unintentional

While it’s true that the change was unintentional at the time, the bugs mentioned in this thread have caused the issue to be carefully reconsidered by the responsible folks at Apple, both from the networking and privacy teams.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Any updates to this original bug report? Mine (Bug # FB9329234) was resolved stating - working as designed.

Can we expect any action to be taken around the additional issue this presents with regards to how iPadOS presents logging information. With the CFNetwork debug profile installed on a device the CFNetworkAgent will advise that the User-Agent Header will be enqueued as part of the request. Without looking at a packet capture collected from an iPadOS device is there any logging information exposed to advise that the user-agent string is not sent as part of the connect request.

@eskimo, we submitted an issue to the feedback assistant with number FB9872194. In general lines, we think that for the most products an alternative to the full user agent could be use the application brand or name inside the user-agent or provide another header field, removing from it the version, the device or another kind of information, that, could be consider a potential risk for the user. In any case, in my opinion, this change affects a large number of companies and should have been done progressively with prior advice.

Thanks for your support @eskimo & @meaton 

This issue also a topic with macOS 12.3 We have a filter at our proxy examing the network traffic for "CFNetwork" or "Macintosh" because macOS generelly sends some requests unauthenticated to identify the requests that are not going to block an ip address. This has worked fine until macOS Big Sur An unauthenticated requests in our company means that the IP address will be blocked for a time for all internet traffic (it is a Cisco recomondation). As far as the user agent is no more present (not only for requests to Apple, also to OneDrive from MS, launchdarkly and opendns/Umbrella) the filter does not work anymore in about 50% of all cases.