With the release of iPadOS 17 we began noticing that the AssetCacheLocator service on iOS/iPadOS was reliably returning a "no caching server" response even when we had active caching servers on the network.
default 2023-09-27 18:40:17.970429 +1000 AssetCacheLocatorService #11553fbd [AssetCacheLocatorService.queue] found no caching servers
This behaviour was also apparent during the beta releases of iPadOS 17 but vanished around beta 8 before returning during the production release.
For a bit of back story our network environment makes use of proxy servers which devices are configured to use via PAC discovery methods. Either via specifying a pac url in config payload or via auto-discovery means via DHCP option 252 or dns. We have been compliant with https PAC file delivery since it was first brought in for the iPadOS 15 release.
After continued testing the behaviour was being cause by a failure in the trust daemon to validate certificates used in SSL handshakes for the AssetCacheLocatorService. This would result in NSURLNetworkError = -999 being reported for network tasks started from the AssetCacheLocatorService.
default 2023-09-27 18:40:17.961617 +1000 AssetCacheLocatorService Task <027E06F3-3994-4BE1-9F3C-EA3C22110ACA>.<1> finished with error [-999] Error Domain=NSURLErrorDomain Code=-999 UserInfo={NSErrorFailingURLStringKey=<private>, NSErrorFailingURLKey=<private>, _NSURLErrorRelatedURLSessionTaskErrorKey=<private>, _NSURLErrorFailingURLSessionTaskErrorKey=<private>, NSLocalizedDescription=<private>}
I initially thought these trust failures were directly related to ocsp and crl requests needed to validate the "https://lcdn-locator.apple.com/lcdn/locate" api. However, when testing was performed utilising no PAC url or via using manual proxy settings AssetCacheLocatorservice would correctly find caching servers on the network.
Puzzled by this response continued analysis of the logs generated from the trust daemon pointed in the direction of HTTPS pac file delivery. It seems that the AssetCacheLocatorService enforces certificate validation for all SSL sessions it establishes. Should a validation fail the entire process is cancelled and a "no caching server" response is returned. The problem is that this is also applying to the PAC file retrieval over HTTPS. I suspect may be causing a loop where it's unable to validate the certificate used for HTTPS pac file retrieval without having the pac file.
After disabling HTTPS pac file support on our network we then observed that the AssetCacheLocatorService completed successfully:
HTTPS PAC File
default 2023-09-27 18:40:17.661202 +1000 AssetCacheLocatorService CFNetwork Diagnostics [3:6] 18:40:17.660 {
Authentication Challenge: (null)
Loader: request POST https://lcdn-locator.apple.com/<redacted>
Challenge: challenge space https://lcdn-locator.apple.com:443/, ServerTrustEvaluationRequested (Hash 91283047ef00ae3c)
} [3:6]
default 2023-09-27 18:40:17.662336 +1000 AssetCacheLocatorService [0x974b0e130] activating connection: mach=true listener=false peer=false name=com.apple.trustd
default 2023-09-27 18:40:17.664988 +1000 AssetCacheLocatorService user sessions enabled, targeting 501
default 2023-09-27 18:40:17.665056 +1000 AssetCacheLocatorService [0x974a255e0] activating connection: mach=true listener=false peer=false name=com.apple.trustd
default 2023-09-27 18:40:17.928484 +1000 AssetCacheLocatorService cleaning up unpooled xpc conn to trustd 0x974a255e0
default 2023-09-27 18:40:17.928567 +1000 AssetCacheLocatorService [0x974a255e0] invalidated on xpc_connection_cancel()
error 2023-09-27 18:40:17.928972 +1000 AssetCacheLocatorService #11553fbd [NSOperationQueue 0x97490ac50 (QOS: UNSPECIFIED)] LocateURL https://lcdn-locator.apple.com/lcdn/locate untrusted: EV=(null) organization=(null)
default 2023-09-27 18:40:17.929114 +1000 AssetCacheLocatorService got event: Connection invalid
default 2023-09-27 18:40:17.929167 +1000 AssetCacheLocatorService Task <027E06F3-3994-4BE1-9F3C-EA3C22110ACA>.<1> auth completion disp=2 cred=0x0
default 2023-09-27 18:40:17.957367 +1000 AssetCacheLocatorService Task <027E06F3-3994-4BE1-9F3C-EA3C22110ACA>.<1> summary for task failure {transaction_duration_ms=3211, response_status=-1, connection=1, reused=1, request_start_ms=0, request_duration_ms=0, response_start_ms=0, response_duration_ms=0, request_bytes=0, response_bytes=0, cache_hit=false}
HTTP Pac File (no https support)
default 2023-09-29 16:30:02.452153 +1000 AssetCacheLocatorService CFNetwork Diagnostics [3:6] 16:30:02.452 {
Authentication Challenge: (null)
Loader: request POST https://lcdn-locator.apple.com/<redacted>
Challenge: challenge space https://lcdn-locator.apple.com:443/, ServerTrustEvaluationRequested (Hash 91283047ef00ae3c)
} [3:6]
default 2023-09-29 16:30:02.454355 +1000 AssetCacheLocatorService [0x81691bab0] activating connection: mach=true listener=false peer=false name=com.apple.trustd
default 2023-09-29 16:30:02.455334 +1000 AssetCacheLocatorService user sessions enabled, targeting 501
default 2023-09-29 16:30:02.455436 +1000 AssetCacheLocatorService [0x816b12dc0] activating connection: mach=true listener=false peer=false name=com.apple.trustd
default 2023-09-29 16:30:02.630496 +1000 AssetCacheLocatorService cleaning up unpooled xpc conn to trustd 0x816b12dc0
default 2023-09-29 16:30:02.630672 +1000 AssetCacheLocatorService [0x816b12dc0] invalidated on xpc_connection_cancel()
default 2023-09-29 16:30:02.631720 +1000 AssetCacheLocatorService got event: Connection invalid
default 2023-09-29 16:30:02.631788 +1000 AssetCacheLocatorService Task <44EC5539-7E06-4BC5-AD2D-F40E43C947B6>.<1> auth completion disp=0 cred=0x816a14490
default 2023-09-29 16:30:02.632041 +1000 AssetCacheLocatorService CFNetwork Diagnostics [3:7] 16:30:02.631 {
Use Credential: (null)
Loader: request POST https://lcdn-locator.apple.com/<redacted>
Credential: Name: iswlocator.apple.com, Persistence: session
} [3:7]
default 2023-09-29 16:30:02.632267 +1000 AssetCacheLocatorService CFNetwork Diagnostics [3:8] 16:30:02.632 {
touchConnection: (null)
Loader: request POST https://lcdn-locator.apple.com/<redacted>
Timeout Interval: 59.739 seconds
} [3:8]
default 2023-09-29 16:30:02.636661 +1000 AssetCacheLocatorService Connection 1: TLS Trust result 0
default 2023-09-29 16:30:02.636774 +1000 AssetCacheLocatorService boringssl_context_evaluate_trust_async_external_block_invoke_3(1631) [C1.1.1.1.1:2][0x816918590] Returning from external verify block with result: true
default 2023-09-29 16:30:02.636935 +1000 AssetCacheLocatorService boringssl_context_certificate_verify_callback(1852) [C1.1.1.1.1:2][0x816918590] Certificate verification result: OK
Can the trust mechanism for this process be adjusted to enforce certificate validation to allow for HTTPS pac file delivery.