intermittent network request timed out on iOS 13.4.1+

[-1001] Error Domain=NSURLErrorDomain Code=-1001 "The request timed out." UserInfo={NSUnderlyingError=0x2834f6d90 {Error Domain=kCFErrorDomainCFNetwork Code=-1001 UserInfo={_kCFStreamErrorCodeKey=-2102, _kCFStreamErrorDomainKey=4}}, NSErrorFailingURLStringKey=<private>, NSErrorFailingURLKey=<private>, _kCFStreamErrorDomainKey=4,


Keep getting this intermittently, request never reaching server. some users getting it quite often


App is build against Xcode 11.1

Intermittent failures to a server can come from a variety of different reasons. Here are a few to debug this type of issue:


1. The most common reason this happens is because your app is inadvertently connecting to a server that you didn't expect to. To debug this ensure that you have knowledge of the network topology you are connecting through and make sure the request is hitting the server you'd expect. To make sure the request is hitting this server, set the access logs to verbose on that server and check them out. This should tell you if the request at least made it to the server and you can debug further server level issues from there.


2. Another common problem relates to number 1, and that is to have knowledge of the network path to and from the downstream server you are talking to. This may not be your case, but some Enterprise networks have several layers of gateways, load balancers, and proxies network requests pass through while traveling to their final destination. If there is an issue at one of these hops this can most certainly cause a request timeout. Look for things gateways and proxies not handling HTTP request / response headers correctly.


3. High volume network concurrency will cause this due to H1 head of line blocking. This may not be your issue, but if you are handling a lot of concurrent network request, let me know and we can dig into this further.


You could dig into this with a packet trace or CFNetwork Diagnostic logging. This should provide more information on the dst addr that is timing out so you would know if you are connecting to the wrong host.



Matt Eaton

DTS Engineering, CoreOS

meaton3 at apple.com

Issue observed in 13.4.1, 13.5 & 13.5.1 as well.


1. The most common reason this happens is because your app is inadvertently connecting to a server that you didn't expect to.  To debug this ensure that you have knowledge of the network topology you are connecting through and make sure the request is hitting the server you'd expect.  To make sure the request is hitting this server, set the access logs to verbose on that server and check them out.  This should tell you if the request at least made it to the server and you can debug further server level issues from there.

We enabled access & error logs on web server but request never reached server but I recieved time out. any reason CFnetwork will block network requests ?


2. Another common problem relates to number 1, and that is to have knowledge of the network path to and from the downstream server you are talking to.  This may not be your case, but some Enterprise networks have several layers of gateways, load balancers, and proxies network requests pass through while traveling to their final destination. If there is an issue at one of these hops this can most certainly cause a request timeout.  Look for things gateways and proxies not handling HTTP request / response headers correctly.

we checked network packets on server. so we have successfull requests network requests all goes through but for failure no request seen


3. High volume network concurrency will cause this due to H1 head of line blocking.  This may not be your issue, but if you are handling a lot of concurrent network request, let me know and we can dig into this further.

There is no difference network load. its below average.


You could dig into this with a packet trace or CFNetwork Diagnostic logging.  This should provide more information on the dst addr that is timing out so you would know if you are connecting to the wrong host.

I have enabled CFNetwork dignostics and trying to reproduce the error but its highly intermittent. while customers are getting it randomly, its taking me day or to reproduce

This is good. If you are not receiving network traffic on the server for the failing requests, but are receiving all other requests, that points to a routing problem in the traffic getting to the server. Try looking at the node directly in front of the server you are trying to hit. If there is nothing in front of the server you are hit or there is no problem there, try hitting a well known endpoint like httpbin to test your API. If you are still seeing problems there, try extracting just your network code to an isolated project.


| We enabled access & error logs on web server but request never reached server but I

| recieved time out. any reason CFnetwork will block network requests ?

|

| we checked network packets on server. so we have successfull requests network requests

| all goes through but for failure no request seen



Matt Eaton

DTS Engineering, CoreOS

meaton3 at apple.com

We implemented retry on -1001 & -1005 errors for time being but in some cases retry request also failing.

below is CFNetwork logs. I was not able to identify any reason for failure even though 1001 is evident below.



Appreciate your help. thank you.





Any update about this topic? We have met the same problem here. Seems user's network is fine and help to check using the Safari, and our app still got time out error, both HTTP and socket connections.

Hi all! Any update on this topic? We have exactly same situation. Some users encountered this network issue.. we don't have any server logs, we can't reproduce it on most of devices; it is random and no logs. Ideas?

Hi Therel! Any update? Same problem here!

+1, this has been happening to us for months. Only occurs when users are on Mobile Data (5G/4G)

Debugging intermittent networking problems is a challenge. How you approach this depends on your app’s audience.

  • If your app ships to a wide range of users, there’s really not much you can do on the debugging front. Rather, you should focus on mitigation strategies, like retries and so on.

  • If your app ships to a smaller group of more technical users, you can ask them to help you debug the issue.

Note that the same app can be in both categories at different times. If you ship a general-purpose apps then you should focus on mitigation in your shipping app and on debugging in the TestFlight build you release to beta testers.

On the debugging front, the weapon of choice here is a sysdiagnose log. For more background on that, see:

One strategy I’ve seen developers use successfully is to add beta-tester-only code to their app that detects the specific problem and, when it sees it, asks the beta tester to trigger a sysdiagnose log and send that to you. That maximises your chances of the sysdiagnose log containing actionable info.


Just to set expectations here, I’ve helped many developers investigate problems like this and the majority of the time the timeouts were caused by network or server problems. That is, iOS is doing the right thing and the root cause is off device. The advantage of this debugging approach is that it lets you divide the problem in half:

  • If the sysdiagnose log suggests an on-device problem, that’s something you can investigate, and potentially discuss with Apple.

  • If it suggests an off-device problem, you can discuss that with your network provider or server folks and then focus your iOS efforts on mitigation.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

intermittent network request timed out on iOS 13.4.1+
 
 
Q