Issue observed in 13.4.1, 13.5 & 13.5.1 as well.
1. The most common reason this happens is because your app is inadvertently connecting to a server that you didn't expect to. To debug this ensure that you have knowledge of the network topology you are connecting through and make sure the request is hitting the server you'd expect. To make sure the request is hitting this server, set the access logs to verbose on that server and check them out. This should tell you if the request at least made it to the server and you can debug further server level issues from there.
We enabled access & error logs on web server but request never reached server but I recieved time out. any reason CFnetwork will block network requests ?
2. Another common problem relates to number 1, and that is to have knowledge of the network path to and from the downstream server you are talking to. This may not be your case, but some Enterprise networks have several layers of gateways, load balancers, and proxies network requests pass through while traveling to their final destination. If there is an issue at one of these hops this can most certainly cause a request timeout. Look for things gateways and proxies not handling HTTP request / response headers correctly.
we checked network packets on server. so we have successfull requests network requests all goes through but for failure no request seen
3. High volume network concurrency will cause this due to H1 head of line blocking. This may not be your issue, but if you are handling a lot of concurrent network request, let me know and we can dig into this further.
There is no difference network load. its below average.
You could dig into this with a packet trace or CFNetwork Diagnostic logging. This should provide more information on the dst addr that is timing out so you would know if you are connecting to the wrong host.
I have enabled CFNetwork dignostics and trying to reproduce the error but its highly intermittent. while customers are getting it randomly, its taking me day or to reproduce