Crash on iOS 14.5：nw_endpoint_flow_copy_path + 44

App & System Services Core OS
carrieni OP
Created Apr ’21
Replies 34
Boosts 0
Views 17k
Participants 20
Hi，
Apps crashed when enter foreground and want to connect with server，since iOS 14.5 released on April 27。

Crash stack like this：

Monitor Type: Mach Exception
Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x0000000000000010
Crashed Thread: 40
Pthread id: 867350
Thread 40 Crashed:
0 libnetwork.dylib nw_endpoint_flow_copy_path + 44
1 libnetwork.dylib nw_endpoint_flow_copy_path + 40
2 libnetwork.dylib nw_endpoint_flow_connected + 320
3 libnetwork.dylib nw_flow_connected + 3668
4 libnetwork.dylib nw_socket_connect + 584
5 libnetwork.dylib nw_endpoint_flow_connect + 164
6 libnetwork.dylib nw_endpoint_flow_setup_protocols + 3272
7 libnetwork.dylib -[NWConcrete_nw_endpoint_flow startWithHandler:] + 3616
8 libnetwork.dylib nw_endpoint_handler_path_change + 10592
9 libnetwork.dylib nw_endpoint_handler_start + 1184
10 libnetwork.dylib __nw_connection_start_block_invoke + 1460
11 imeituan __innerBlockOnce_block_invoke (SCRCrashMonitor_GCD.mm:191)
12 libdispatch.dylib _dispatch_call_block_and_release + 32
13 libdispatch.dylib _dispatch_client_callout + 20
14 libdispatch.dylib _dispatch_lane_serial_drain + 620
15 libdispatch.dylib _dispatch_lane_invoke + 456
16 libdispatch.dylib _dispatch_workloop_invoke + 1680
17 libdispatch.dylib _dispatch_workloop_worker_thread + 764
18 libsystem_pthread.dylib _pthread_wqthread + 276
Enqueue thread backtrace:
0 imeituan innerBlockOnce (SCRCrashMonitor_GCD.mm:0)
1 imeituan _dispatch_async (SCRCrashMonitor_GCD.mm:0)
2 libnetwork.dylib nw_connection_start + 280
3 libnetwork.dylib tcp_connection_start + 1496
4 CFNetwork 0x0000000180b7c3bc + 168
5 CFNetwork 0x0000000180b79b80 + 2776
6 CFNetwork 0x0000000180b77918 + 1724
7 CoreFoundation _CFStreamOpen + 140
8 imeituan -[GCDAsyncSocket openStreams] (GCDAsyncSocket.m:6998)
9 imeituan __29-[GCDAsyncSocket didConnect:]_block_invoke330 (GCDAsyncSocket.m:2369)
10 imeituan __29-[GCDAsyncSocket didConnect:]_block_invoke_2 (GCDAsyncSocket.m:2396)
11 imeituan __innerBlockOnce_block_invoke (SCRCrashMonitor_GCD.mm:191)
12 libdispatch.dylib _dispatch_call_block_and_release + 32
13 libdispatch.dylib _dispatch_client_callout + 20
14 libdispatch.dylib _dispatch_lane_serial_drain + 620
15 libdispatch.dylib _dispatch_lane_invoke + 404
16 libdispatch.dylib _dispatch_workloop_worker_thread + 764
17 libsystem_pthread.dylib _pthread_wqthread + 276
18 libsystem_pthread.dylib start_wqthread + 8
Replies  34
Boosts  0
Views  17k
Participants  20
DTS Engineer OP
Apple
Apr ’21

Apps crashed when enter foreground and want to connect with server

Please post a full Apple crash report. Use the text attachment feature (the paperclip icon) to avoid clogging up the timeline.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"
averagejjjjjjoe OP
Apr ’21
We are experiencing this as well, #1 crash in iOS 14.5 devices in our app. Same stacktrace in the crashed thread.
naikunwang OP
Apr ’21
i have same crash , how can i fix it ?
d@n OP
May ’21
same crash reported,,, iOS 14.5 and 14.6beta
Adithya Reddy OP
May ’21
Crashed: com.apple.network.connection-65
EXC_BAD_ACCESS KERN_INVALID_ADDRESS 0x0000000000000010
nw_endpoint_flow_copy_path

experiencing the same crash as well.
iOS versions (14.5 and 14.6)

Soaring since April 27th.
DTS Engineer OP
Apple
May ’21
averagejjjjjjoe, naikunwang, d@n, Adithya Reddy, per my initial response, I’m not going to be able to provide any further insight here without an Apple crash report. If you have one, please post it.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"
autobotCA OP
May ’21
Here is a crash report

CrashReport
autobotCA OP
May ’21
Here is a second crash report from a more modern version
Crashreport 2


DTS Engineer OP
Apple
May ’21
Thanks for the crash reports. These were enough to point me to a bug (r. 71651938) that we’re using to track this crash. I don’t have a lot of info to share but I can say that:

  • There are a wide range of apps experiencing the problem and thus it’s most likely a problem in the OS rather than a problem with your code.

  • Based on our analysis so far this seems to be an iOS 14.5 regression.

Normally I’d recommend that you file your own bug about this, with whatever extra info you have to help the Network framework folks debug it, but in this case I happen to know (via other channels) that you’ve already done that.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"
Wingzki OP
May ’21
We got more information about this.

2021-05-07 14:09:31.538353+0800 Hello[1986:2515251] [] nw_path_evaluator_create_flow_inner NECP_CLIENT_ACTION_ADD_FLOW 0A158B95-F5ED-4E9B-8D68-4BDC211744F9 [48: Address already in use]
2021-05-07 14:09:31.538839+0800 Hello[1986:2515251] [connection] nw_endpoint_flow_setup_channel [C16.1 123.125.102.213:5222 in_progress channel-flow (satisfied (Path is satisfied), interface: en0, scoped, ipv4, dns)] failed to request add nexus flow
2021-05-07 14:09:31.541714+0800 Hello[1986:2515251] [] nw_path_evaluator_create_flow_inner NECP_CLIENT_ACTION_ADD_FLOW 81D30F6B-0642-49B1-A0F4-C756CA26B1F9 [48: Address already in use]
2021-05-07 14:09:31.541917+0800 Hello[1986:2515251] [connection] nw_endpoint_flow_setup_channel [C16.2 123.125.102.213:5222 in_progress channel-flow (satisfied (Path is satisfied), interface: en0, scoped, ipv4, dns)] failed to request add nexus flow
2021-05-07 14:09:31.543370+0800 Hello[1986:2515251] [] nw_socket_copy_fd_wrapper called with null protocol
2021-05-07 14:09:31.546335+0800 Hello[1986:2515251] [] nw_socket_copy_fd_wrapper called with null protocol, dumping


Thread 6 Queue : com.apple.network.connection-16
carrieni OP
May ’21
What is the reason? How to fix it? eskimo 
Crash Report


DTS Engineer OP
Apple
May ’21

What is the reason?

Please read my previous response. If anything in that response was unclear, feel free to post a specific question that references it.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"
sukov OP
May ’21
hi @eskimo.
I read your response but do you have any workaround solution for the app side?
It's hard for us if nothing to do with this crash

Thanks
a20251313 OP
May ’21
Crashed:
nw_endpoint_flow_copy_path

experiencing the same crash as well. 
iOS versions (14.5 and 14.5.1)

Soaring since April 27th.

Over 7000 devices one day.
DTS Engineer OP
Apple
May ’21

do you have any workaround solution for the app side?

Quick question: Are you using a third-party library for your networking? Perhaps GCDAsyncSocket?

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"
aakarshsasi OP
May ’21

Quick question: Are you using a third-party library for your networking? Perhaps GCDAsyncSocket?

Yes, we do
carrieni OP
May ’21
We use GCDAsyncSocket too.
DTS Engineer OP
Apple
May ’21

We use GCDAsyncSocket too.

OK. Let me start by being very clear about this: This is Apple’s bug to fix. GCDAsyncSocket is not doing anything wrong here, it just happens to be using the system in a way that triggers this bug.

Having said that, moving to Network framework would be a good workaround for this bug. There’s actually multiple reasons for this:

  • It works around this bug.

  • GCDAsyncSocket is based on APIs that are effectively deprecated [1].

  • In most cases GCDAsyncSocket users can migrate directly to Network framework without the need for any wrapper.

  • Network framework is the future of (non-HTTP) networking on our platforms. That’s where our efforts are being placed, and now that we’ve caught up with the basics this is likely to show more and more. As an example, folks using Network framework automatically get TLS 1.3 support; folks using older APIs, like CFSocketStream and BSD Sockets with Secure Transport, are stuck on TLS 1.2.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] It actually uses two different APIs, CFSocketStream and BSD Sockets. CFSocketStream is definitely on the path to formal deprecation. I can’t imagine us ever formally deprecating BSD Sockets but it is very much a legacy API.
carrieni OP
May ’21
Will it be fixed in 14.6 ？ @eskimo
DTS Engineer OP
Apple
May ’21

Will it be fixed in 14.6 ？

I am, alas, unable to predict the future.

I can say that it’s not fixed in the current 14.6rc (18F71). It also not fixed in the 14.7b1 (18G5023c) that we just started seeding. I recommend that you continue testing with 14.7 betas as they are released.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"
K.H OP
May ’21
We have the same problem, This is a relatively serious problem by analyze our monitoring data. It caused a very large amount of crash and make user experience bad. If it is Apple’s bug, Why not fix it as soon as possible, moving to Network framework in a short time is not too easy for a system like base on GCDAsyncSocket.And there's not enough introduction to do it :(
ligaofeng OP
May ’21

Will it be fixed in 14.7 ？ @eskimo
Adithya Reddy OP
May ’21

@eskimo

  1. Moving to Network.framework on such short notice is quite hard. iOS14.5 adoption only increasing.
  2. Network.framework doesn't support iOS 11 (yes, we do have a chunk of users on iOS 11).

Please suggest if there's any other workaround here that Network team can think of.

DTS Engineer OP
Apple
May ’21

ligaofeng wrote:

Will it be fixed in 14.7 ？

I am, alas, unable to predict the future. I do, however, recommend that you continue to test with beta release systems as they become available.

Adithya Reddy wrote:

Moving to Network.framework on such short notice is quite hard.

Understood. I only suggested it because a) it will work around this problem, and b) it’s the right thing to do going forward.

Please suggest if there's any other workaround here

Not that I’m aware of. Well, there are other things you could (BSD Sockets, anyone?) but nothing that’s easier than moving to Network framework, which you’ve already ruled out.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

DTS Engineer OP
Apple
Jun ’21

We believe that the just-seeded iOS 14.7 beta 2 (18G5033e) has fixed the bug being discussed in this thread (r. 71651938). Please try it out, and let us know what you see.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Crash on iOS 14.5：nw_endpoint_flow_copy_path + 44
First post date Last post date
 
 
Q