Crash in com.apple.CFNetwork.addPersistCacheToStorageDaemon

I logged this as rdar://26599310 but I also wanted to see if anyone else has seen something similar. I'm seeing a fem different vairations of this crash where the dispatch queue com.apple.CFNetwork.addPersistCacheToStorageDaemon is crashing due to some memory problem. I see SIGSEGVs and NSSecureCoding exceptions when this queue is creating instances of NSXPCConnections. From what I gathered, it might be a race condition when an item is being added to the cache or the cache is emptying during or shortly after a SSL handshake. While I'm still using RestKit (so AFnetworking underneath -> NSURLConnection), the only thing that I think might be out of the ordinary is that I also have a custom NSURLProtocol that handles mutual TLS authentication challenges.


Example:

Incident Identifier: 6A3D3C23-3C5E-4A4E-9793-C24BFB8B9492
CrashReporter Key:   f53b85236e7bd2192f8e426c311e2ad7000f7e3c
Hardware Model:      iPad4,1
Process:             MyApp [1741]
Path:                /private/var/mobile/Containers/Bundle/Application/9C4AD9FE-3A9E-4336-9BDF-FAD81CAC744D/MyApp.app/MyApp
Identifier:          com.mycompany.myapp
Version:             4.2 (4.2)
Code Type:           ARM-64 (Native)
Parent Process:      launchd [1]

Date/Time:
2016-05-20 14:03:55.55 -0500
Launch Time:
2016-05-20 14:01:29.29 -0500
OS Version:
iOS 9.1 (13B143)
Report Version:
105

Exception Type:  EXC_CRASH (SIGSEGV)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Exception Note:  EXC_CORPSE_NOTIFY
Triggered by Thread:  0

Filtered syslog:
None found

Global Trace Buffer (reverse chronological seconds):
4.449833     AppleJPEG                     0x000000018a4d9004 [0x12fa46000] Decoding completed without errors
4.449956     AppleJPEG                     0x000000018a4d719c [0x12fa46000] Options: 48x64 [FFFFFFFF,FFFFFFFF] 00025060
4.449956     AppleJPEG                     0x000000018a4d7054 [0x12fa46000] Decoding: C0 0x00300040 0x0000304A 0x22111100 0x00000000 2100
4.457737     AppleJPEG                     0x000000018a4d6348 [0x12fa46000] Created session
4.491343     CFNetwork                     0x000000018290da18 TCP Conn 0x1313242e0 SSL Handshake DONE
4.518652     CFNetwork                     0x000000018290d928 TCP Conn 0x1313242e0 starting SSL negotiation
4.519018     CFNetwork                     0x00000001829afa30 TCP Conn 0x1313242e0 complete. fd: 21, err: 0
4.519477     CFNetwork                     0x00000001829b0f5c TCP Conn 0x1313242e0 event 1. err: 0
4.532374     CFNetwork                     0x00000001829b1034 TCP Conn 0x1313242e0 started
6.785654     CFNetwork                     0x000000018290da18 TCP Conn 0x131109530 SSL Handshake DONE
6.786804     CFNetwork                     0x000000018290da18 TCP Conn 0x131236270 SSL Handshake DONE
6.799260     CFNetwork                     0x000000018290d928 TCP Conn 0x131109530 starting SSL negotiation
6.799762     CFNetwork                     0x000000018290d928 TCP Conn 0x131236270 starting SSL negotiation
6.799858     CFNetwork                     0x00000001829afa30 TCP Conn 0x131109530 complete. fd: 15, err: 0
6.800533     CFNetwork                     0x00000001829afa30 TCP Conn 0x131236270 complete. fd: 16, err: 0
6.800803     CFNetwork                     0x00000001829b0f5c TCP Conn 0x131109530 event 1. err: 0
6.801410     CFNetwork                     0x00000001829b0f5c TCP Conn 0x131236270 event 1. err: 0
6.814756     CFNetwork                     0x00000001829b1034 TCP Conn 0x131236270 started
6.814900     CFNetwork                     0x00000001829b1034 TCP Conn 0x131109530 started
119.099618   CFNetwork                     0x000000018290da18 TCP Conn 0x130be18f0 SSL Handshake DONE
119.198016   CFNetwork                     0x000000018290d928 TCP Conn 0x130be18f0 starting SSL negotiation
119.198400   CFNetwork                     0x00000001829afa30 TCP Conn 0x130be18f0 complete. fd: 34, err: 0
119.199925   CFNetwork                     0x00000001829b0f5c TCP Conn 0x130be18f0 event 1. err: 0
119.228514   CFNetwork                     0x00000001829b1034 TCP Conn 0x130be18f0 started
119.262970   CFNetwork                     0x000000018290da18 TCP Conn 0x130c45650 SSL Handshake DONE
119.269396   CFNetwork                     0x000000018290da18 TCP Conn 0x130c4a780 SSL Handshake DONE
119.285508   CFNetwork                     0x000000018290da18 TCP Conn 0x130c4b2d0 SSL Handshake DONE
119.560258   CFNetwork                     0x000000018290d928 TCP Conn 0x130c4a780 starting SSL negotiation
119.560513   CFNetwork                     0x00000001829afa30 TCP Conn 0x130c4a780 complete. fd: 29, err: 0
119.561818   CFNetwork                     0x00000001829b0f5c TCP Conn 0x130c4a780 event 1. err: 0

Thread 0 name:  Dispatch queue: com.apple.main-thread
Thread 0 Crashed:
0   libsystem_kernel.dylib            0x00000001986a0a40 mach_msg_trap + 8
1   libsystem_kernel.dylib            0x00000001986a08bc mach_msg + 72
2   CoreFoundation                    0x00000001831c4108 __CFRunLoopServiceMachPort + 196
3   CoreFoundation                    0x00000001831c1e0c __CFRunLoopRun + 1032
4   CoreFoundation                    0x00000001830f0ca0 CFRunLoopRunSpecific + 384
5   GraphicsServices                  0x000000018e158088 GSEventRunModal + 180
6   UIKit                             0x0000000188808ffc UIApplicationMain + 204
7   MyApp                             0x00000001000b9834 main (main.m:16)
8   libdyld.dylib                     0x000000019859e8b8 start + 4

Thread 1 name:  Dispatch queue: com.apple.libdispatch-manager
Thread 1:
0   libsystem_kernel.dylib            0x00000001986bc4fc kevent_qos + 8
1   libdispatch.dylib                 0x0000000198580a04 _dispatch_mgr_invoke + 232
2   libdispatch.dylib                 0x000000019856f874 _dispatch_source_invoke + 0

Thread 2 name:  WebThread
Thread 2:
0   libsystem_kernel.dylib            0x00000001986a0a40 mach_msg_trap + 8
1   libsystem_kernel.dylib            0x00000001986a08bc mach_msg + 72
2   CoreFoundation                    0x00000001831c4108 __CFRunLoopServiceMachPort + 196
3   CoreFoundation                    0x00000001831c1e0c __CFRunLoopRun + 1032
4   CoreFoundation                    0x00000001830f0ca0 CFRunLoopRunSpecific + 384
5   WebCore                           0x0000000194f4654c RunWebThread(void*) + 456
6   libsystem_pthread.dylib           0x000000019878fb28 _pthread_body + 156
7   libsystem_pthread.dylib           0x000000019878fa8c _pthread_body + 0
8   libsystem_pthread.dylib           0x000000019878d028 thread_start + 4

Thread 3 name:  com.apple.NSURLConnectionLoader
Thread 3:
0   libsystem_kernel.dylib            0x00000001986a0a40 mach_msg_trap + 8
1   libsystem_kernel.dylib            0x00000001986a08bc mach_msg + 72
2   CoreFoundation                    0x00000001831c4108 __CFRunLoopServiceMachPort + 196
3   CoreFoundation                    0x00000001831c1e0c __CFRunLoopRun + 1032
4   CoreFoundation                    0x00000001830f0ca0 CFRunLoopRunSpecific + 384
5   CFNetwork                         0x0000000182989b84 +[NSURLConnection(Loader) _resourceLoadLoop:] + 412
6   Foundation                        0x000000018414fc80 __NSThread__start__ + 1000
7   libsystem_pthread.dylib           0x000000019878fb28 _pthread_body + 156
8   libsystem_pthread.dylib           0x000000019878fa8c _pthread_body + 0
9   libsystem_pthread.dylib           0x000000019878d028 thread_start + 4

Thread 4 name:  com.apple.CFSocket.private
Thread 4:
0   libsystem_kernel.dylib            0x00000001986bb368 __select + 8
1   CoreFoundation                    0x00000001831ca670 __CFSocketManager + 648
2   libsystem_pthread.dylib           0x000000019878fb28 _pthread_body + 156
3   libsystem_pthread.dylib           0x000000019878fa8c _pthread_body + 0
4   libsystem_pthread.dylib           0x000000019878d028 thread_start + 4

Thread 5 name:  JavaScriptCore::Marking
Thread 5:
0   libsystem_kernel.dylib            0x00000001986baf48 __psynch_cvwait + 8
1   libsystem_pthread.dylib           0x000000019878ece8 _pthread_cond_wait + 648
2   libc++.1.dylib                    0x0000000197398074 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 56
3   JavaScriptCore                    0x0000000184b6cad8 JSC::GCThread::waitForNextPhase() + 144
4   JavaScriptCore                    0x0000000184b6cb70 JSC::GCThread::gcThreadMain() + 84
5   JavaScriptCore                    0x0000000184842824 ***::threadEntryPoint(void*) + 212
6   JavaScriptCore                    0x0000000184842734 ***::wtfThreadEntryPoint(void*) + 24
7   libsystem_pthread.dylib           0x000000019878fb28 _pthread_body + 156
8   libsystem_pthread.dylib           0x000000019878fa8c _pthread_body + 0
9   libsystem_pthread.dylib           0x000000019878d028 thread_start + 4

Thread 6 name:  AVAudioSession Notify Thread
Thread 6:
0   libsystem_kernel.dylib            0x00000001986a0a40 mach_msg_trap + 8
1   libsystem_kernel.dylib            0x00000001986a08bc mach_msg + 72
2   CoreFoundation                    0x00000001831c4108 __CFRunLoopServiceMachPort + 196
3   CoreFoundation                    0x00000001831c1e0c __CFRunLoopRun + 1032
4   CoreFoundation                    0x00000001830f0ca0 CFRunLoopRunSpecific + 384
5   libAVFAudio.dylib                 0x0000000181859810 GenericRunLoopThread::Entry(void*) + 164
6   libAVFAudio.dylib                 0x000000018182e384 CAPThread::Entry(CAPThread*) + 84
7   libsystem_pthread.dylib           0x000000019878fb28 _pthread_body + 156
8   libsystem_pthread.dylib           0x000000019878fa8c _pthread_body + 0
9   libsystem_pthread.dylib           0x000000019878d028 thread_start + 4

Thread 7 name:  com.apple.coremedia.player.async
Thread 7:
0   libsystem_kernel.dylib            0x00000001986a0a7c semaphore_wait_trap + 8
1   libdispatch.dylib                 0x000000019857e614 _dispatch_semaphore_wait_slow + 244
2   MediaToolbox                      0x0000000185a24aa4 fpa_AsyncMovieControlThread + 1948
3   CoreMedia                         0x0000000183a76a70 figThreadMain + 272
4   libsystem_pthread.dylib           0x000000019878fb28 _pthread_body + 156
5   libsystem_pthread.dylib           0x000000019878fa8c _pthread_body + 0
6   libsystem_pthread.dylib           0x000000019878d028 thread_start + 4

Thread 8 name:  WebCore: CFNetwork Loader
Thread 8:
0   libsystem_kernel.dylib            0x00000001986a0a40 mach_msg_trap + 8
1   libsystem_kernel.dylib            0x00000001986a08bc mach_msg + 72
2   CoreFoundation                    0x00000001831c4108 __CFRunLoopServiceMachPort + 196
3   CoreFoundation                    0x00000001831c1e0c __CFRunLoopRun + 1032
4   CoreFoundation                    0x00000001830f0ca0 CFRunLoopRunSpecific + 384
5   WebCore                           0x0000000194f73e7c WebCore::runLoaderThread(void*) + 272
6   JavaScriptCore                    0x0000000184842824 ***::threadEntryPoint(void*) + 212
7   JavaScriptCore                    0x0000000184842734 ***::wtfThreadEntryPoint(void*) + 24
8   libsystem_pthread.dylib           0x000000019878fb28 _pthread_body + 156
9   libsystem_pthread.dylib           0x000000019878fa8c _pthread_body + 0
10  libsystem_pthread.dylib           0x000000019878d028 thread_start + 4

Thread 9 name:  AFNetworking
Thread 9:
0   libsystem_kernel.dylib            0x00000001986a0a40 mach_msg_trap + 8
1   libsystem_kernel.dylib            0x00000001986a08bc mach_msg + 72
2   CoreFoundation                    0x00000001831c4108 __CFRunLoopServiceMachPort + 196
3   CoreFoundation                    0x00000001831c1e0c __CFRunLoopRun + 1032
4   CoreFoundation                    0x00000001830f0ca0 CFRunLoopRunSpecific + 384
5   Foundation                        0x00000001840692bc -[NSRunLoop(NSRunLoop) runMode:beforeDate:] + 308
6   Foundation                        0x00000001840be8f4 -[NSRunLoop(NSRunLoop) run] + 88
7   MyApp                             0x00000001008a96ec +[AFURLConnectionOperation networkRequestThreadEntryPoint:] (AFURLConnectionOperation.m:190)
8   Foundation                        0x000000018414fc80 __NSThread__start__ + 1000
9   libsystem_pthread.dylib           0x000000019878fb28 _pthread_body + 156
10  libsystem_pthread.dylib           0x000000019878fa8c _pthread_body + 0
11  libsystem_pthread.dylib           0x000000019878d028 thread_start + 4

Thread 10 name:  com.apple.coreaudio.AQClient
Thread 10:
0   libsystem_kernel.dylib            0x00000001986a0a40 mach_msg_trap + 8
1   libsystem_kernel.dylib            0x00000001986a08bc mach_msg + 72
2   CoreFoundation                    0x00000001831c4108 __CFRunLoopServiceMachPort + 196
3   CoreFoundation                    0x00000001831c1e0c __CFRunLoopRun + 1032
4   CoreFoundation                    0x00000001830f0ca0 CFRunLoopRunSpecific + 384
5   AudioToolbox                      0x00000001823ce808 GenericRunLoopThread::Entry(void*) + 164
6   AudioToolbox                      0x00000001823c0ba4 CAPThread::Entry(CAPThread*) + 124
7   libsystem_pthread.dylib           0x000000019878fb28 _pthread_body + 156
8   libsystem_pthread.dylib           0x000000019878fa8c _pthread_body + 0
9   libsystem_pthread.dylib           0x000000019878d028 thread_start + 4

Thread 11:
0   libsystem_kernel.dylib            0x00000001986baf48 __psynch_cvwait + 8
1   libsystem_pthread.dylib           0x000000019878ece8 _pthread_cond_wait + 648
2   MyApp                             0x000000010054a618 0x1000b4000 + 4810264
3   MyApp                             0x00000001006f3a70 0x1000b4000 + 6552176
4   MyApp                             0x00000001005e6058 0x1000b4000 + 5447768
5   MyApp                             0x00000001005e62ac 0x1000b4000 + 5448364
6   MyApp                             0x00000001004d3ccc 0x1000b4000 + 4324556
7   MyApp                             0x000000010054a660 0x1000b4000 + 4810336
8   Foundation                        0x000000018414fc80 __NSThread__start__ + 1000
9   libsystem_pthread.dylib           0x000000019878fb28 _pthread_body + 156
10  libsystem_pthread.dylib           0x000000019878fa8c _pthread_body + 0
11  libsystem_pthread.dylib           0x000000019878d028 thread_start + 4

Thread 12:
0   libsystem_kernel.dylib            0x00000001986baf48 __psynch_cvwait + 8
1   libsystem_pthread.dylib           0x000000019878ece8 _pthread_cond_wait + 648
2   MyApp                             0x000000010054a618 0x1000b4000 + 4810264
3   MyApp                             0x00000001006f3a70 0x1000b4000 + 6552176
4   MyApp                             0x00000001005e6058 0x1000b4000 + 5447768
5   MyApp                             0x00000001004d2070 0x1000b4000 + 4317296
6   MyApp                             0x00000001005e55f4 0x1000b4000 + 5445108
7   MyApp                             0x000000010064532c 0x1000b4000 + 5837612
8   MyApp                             0x000000010054a660 0x1000b4000 + 4810336
9   Foundation                        0x000000018414fc80 __NSThread__start__ + 1000
10  libsystem_pthread.dylib           0x000000019878fb28 _pthread_body + 156
11  libsystem_pthread.dylib           0x000000019878fa8c _pthread_body + 0
12  libsystem_pthread.dylib           0x000000019878d028 thread_start + 4

Thread 13:
0   libsystem_kernel.dylib            0x00000001986baf48 __psynch_cvwait + 8
1   libsystem_pthread.dylib           0x000000019878ece8 _pthread_cond_wait + 648
2   MyApp                             0x000000010054a618 0x1000b4000 + 4810264
3   MyApp                             0x000000010070d190 0x1000b4000 + 6656400
4   MyApp                             0x0000000100709acc 0x1000b4000 + 6642380
5   MyApp                             0x000000010069c800 0x1000b4000 + 6195200
6   MyApp                             0x0000000100510dc8 0x1000b4000 + 4574664
7   MyApp                             0x000000010054a660 0x1000b4000 + 4810336
8   Foundation                        0x000000018414fc80 __NSThread__start__ + 1000
9   libsystem_pthread.dylib           0x000000019878fb28 _pthread_body + 156
10  libsystem_pthread.dylib           0x000000019878fa8c _pthread_body + 0
11  libsystem_pthread.dylib           0x000000019878d028 thread_start + 4

Thread 14:
0   libsystem_kernel.dylib            0x00000001986bbb6c __workq_kernreturn + 8
1   libsystem_pthread.dylib           0x000000019878d530 _pthread_wqthread + 1284
2   libsystem_pthread.dylib           0x000000019878d020 start_wqthread + 4

Thread 15:
0   libsystem_kernel.dylib            0x00000001986bbb6c __workq_kernreturn + 8
1   libsystem_pthread.dylib           0x000000019878d530 _pthread_wqthread + 1284
2   libsystem_pthread.dylib           0x000000019878d020 start_wqthread + 4

Thread 16:
0   libsystem_kernel.dylib            0x00000001986bbb6c __workq_kernreturn + 8
1   libsystem_pthread.dylib           0x000000019878d530 _pthread_wqthread + 1284
2   libsystem_pthread.dylib           0x000000019878d020 start_wqthread + 4

Thread 17:
0   libsystem_kernel.dylib            0x00000001986bbb6c __workq_kernreturn + 8
1   libsystem_pthread.dylib           0x000000019878d530 _pthread_wqthread + 1284
2   libsystem_pthread.dylib           0x000000019878d020 start_wqthread + 4

Thread 18:
0   libsystem_kernel.dylib            0x00000001986bbb6c __workq_kernreturn + 8
1   libsystem_pthread.dylib           0x000000019878d530 _pthread_wqthread + 1284
2   libsystem_pthread.dylib           0x000000019878d020 start_wqthread + 4

Thread 19:
0   libsystem_kernel.dylib            0x00000001986bbb6c __workq_kernreturn + 8
1   libsystem_pthread.dylib           0x000000019878d530 _pthread_wqthread + 1284
2   libsystem_pthread.dylib           0x000000019878d020 start_wqthread + 4

Thread 20 name:  Dispatch queue: com.apple.CFNetwork.addPersistCacheToStorageDaemon
Thread 20:
0   libsystem_platform.dylib          0x0000000198786220 _platform_memmove + 48
1   Foundation                        0x00000001840c2138 NSCopyMemoryPages + 72
2   Foundation                        0x000000018409dae8 -[NSData(NSData) getBytes:length:] + 132
3   Foundation                        0x00000001840c8db4 -[NSData(NSData) replacementObjectForCoder:] + 140
4   Foundation                        0x000000018406148c -[NSXPCEncoder _replaceObject:] + 116
5   Foundation                        0x00000001840cb5e4 -[NSXPCEncoder _encodeArrayOfObjects:forKey:] + 228
6   Foundation                        0x00000001840c8544 -[NSDictionary(NSDictionary) encodeWithCoder:] + 936
7   Foundation                        0x0000000184061e30 -[NSXPCEncoder _encodeObject:] + 684
8   Foundation                        0x0000000184062318 encodeInvocationArguments + 320
9   Foundation                        0x0000000184061fd8 -[NSXPCEncoder encodeInvocation:] + 368
10  Foundation                        0x0000000184061e30 -[NSXPCEncoder _encodeObject:] + 684
11  Foundation                        0x0000000184237198 -[NSXPCConnection _sendInvocation:withProxy:remoteInterface:withErrorHandler:timeout:userInfo:] + 2328
12  CoreFoundation                    0x0000000183210a30 ___forwarding___ + 408
13  CoreFoundation                    0x0000000183114cac _CF_forwarding_prep_0 + 92
14  CFNetwork                         0x000000018295db78 ___ZN12__CFURLCache23CreateAndStoreCacheNodeEP16__CFURLCacheNodePK20_CFCachedURLResponsePK10__CFStringPK13_CFURLRequestPKvbRb_block_invoke + 1964
15  libdispatch.dylib                 0x000000019856d6e8 _dispatch_call_block_and_release + 24
16  libdispatch.dylib                 0x000000019856d6a8 _dispatch_client_callout + 16
17  libdispatch.dylib                 0x00000001985796ec _dispatch_queue_drain + 864
18  libdispatch.dylib                 0x00000001985711ac _dispatch_queue_invoke + 464
19  libdispatch.dylib                 0x000000019857b5bc _dispatch_root_queue_drain + 728
20  libdispatch.dylib                 0x000000019857b2dc _dispatch_worker_thread3 + 112
21  libsystem_pthread.dylib           0x000000019878d470 _pthread_wqthread + 1092
22  libsystem_pthread.dylib           0x000000019878d020 start_wqthread + 4

Thread 21:
0   libsystem_kernel.dylib            0x00000001986baf48 __psynch_cvwait + 8
1   libsystem_pthread.dylib           0x000000019878ece8 _pthread_cond_wait + 648
2   libc++.1.dylib                    0x0000000197398124 std::__1::condition_variable::__do_timed_wait(std::__1::unique_lock<std::__1::mutex>&, std::__1::chrono::time_point<std::__1::chrono::system_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >) + 140
3   JavaScriptCore                    0x0000000184dc0068 std::__1::cv_status std::__1::condition_variable::wait_until<std::__1::chrono::steady_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >(std::__1::unique_lock<std::__1::mutex>&, std::__1::chrono::time_point<std::__1::chrono::steady_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > > const&) + 136
4   JavaScriptCore                    0x0000000184dbff2c std::__1::cv_status std::__1::condition_variable_any::wait_until<std::__1::unique_lock<bmalloc::Mutex>, std::__1::chrono::steady_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >(std::__1::unique_lock<bmalloc::Mutex>&, std::__1::chrono::time_point<std::__1::chrono::steady_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > > const&) + 124
5   JavaScriptCore                    0x0000000184dbfe68 bmalloc::AsyncTask<bmalloc::Heap, void (bmalloc::Heap::*)()>::entryPoint() + 192
6   JavaScriptCore                    0x0000000184dbfd9c bmalloc::AsyncTask<bmalloc::Heap, void (bmalloc::Heap::*)()>::pthreadEntryPoint(void*) + 12
7   libsystem_pthread.dylib           0x000000019878fb28 _pthread_body + 156
8   libsystem_pthread.dylib           0x000000019878fa8c _pthread_body + 0
9   libsystem_pthread.dylib           0x000000019878d028 thread_start + 4

Thread 22 name:  WebCore: LocalStorage
Thread 22:
0   libsystem_kernel.dylib            0x00000001986baf48 __psynch_cvwait + 8
1   libsystem_pthread.dylib           0x000000019878ece8 _pthread_cond_wait + 648
2   JavaScriptCore                    0x0000000184845d90 ***::ThreadCondition::timedWait(***::Mutex&, double) + 80
3   WebKitLegacy                      0x0000000195f4f07c std::__1::unique_ptr<std::__1::function<void ()>, std::__1::default_delete<std::__1::function<void ()> > > ***::MessageQueue<std::__1::function<void ()> >::waitForMessageFilteredWithTimeout<***::MessageQueue<std::__1::function<void ()> >::waitForMessage()::'lambda'(std::__1::function<void ()> const&)>(***::MessageQueueWaitResult&, ***::MessageQueue<std::__1::function<void ()> >::waitForMessage()::'lambda'(std::__1::function<void ()> const&)&&, double) + 88
4   WebKitLegacy                      0x0000000195f4e5e4 WebCore::StorageThread::threadEntryPoint() + 68
5   JavaScriptCore                    0x0000000184842824 ***::threadEntryPoint(void*) + 212
6   JavaScriptCore                    0x0000000184842734 ***::wtfThreadEntryPoint(void*) + 24
7   libsystem_pthread.dylib           0x000000019878fb28 _pthread_body + 156
8   libsystem_pthread.dylib           0x000000019878fa8c _pthread_body + 0
9   libsystem_pthread.dylib           0x000000019878d028 thread_start + 4

Thread 0 crashed with ARM Thread State (64-bit):
    x0: 0x0000000010004005   x1: 0x0000000007000806   x2: 0x0000000000000000   x3: 0x0000000000000c00
    x4: 0x0000000000001b03   x5: 0x00000000ffffffff   x6: 0x0000000000000000   x7: 0x000000016fd49370
    x8: 0x00000000fffffbbf   x9: 0x0000000007000000  x10: 0x0000000007000100  x11: 0x0000000000067e9f
   x12: 0x0000000000000001  x13: 0x0036b1000036b203  x14: 0x0000000000000000  x15: 0x0036b2000036b200
   x16: 0xffffffffffffffe1  x17: 0x0000000000000000  x18: 0x0000000000000000  x19: 0x0000000000000000
   x20: 0x00000000ffffffff  x21: 0x0000000000001b03  x22: 0x0000000000000c00  x23: 0x000000016fd4ad38
   x24: 0x0000000007000806  x25: 0x0000000000000000  x26: 0x0000000007000806  x27: 0x0000000000000c00
   x28: 0x0000000000000001  fp: 0x000000016fd4ac40   lr: 0x00000001986a08bc
    sp: 0x000000016fd4abf0   pc: 0x00000001986a0a40 cpsr: 0x60000000

Was that crash log generated by the Apple crash reporter? Or a third-party crash reporter?

The reason I ask is that it seems non-sensical. I agree with your analysis that the likely cause of the crash is thread 20, but the crash log says that thread 0 crashed. You sometimes see a disparity like this when dealing with asynchronous crash sources, but in this case the crash source is a

SIGSEGV
, which is most definitely synchronous. The only explanation for this that I can think of is that you have a non-Apple crash reporter in the loop here. Or perhaps you have some other reason to be catching and re-throwing
SIGSEGV
.

Anyway, this stuff matters because the thread state shown in the crash log is for thread 0, which isn’t very useful if it’s actually thread 20 that crashed.

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

WWDC runs Mon, 13 Jun through to Fri, 17 Jun. During that time all of DTS will be at the conference, helping folks out face-to-face. http://developer.apple.com/wwdc/

The crash log was pulled from a device via Xcode. In the application, I do have an uncaught exception handler and signal handler to generate crash reports and send them to Splunk. Specifically, the code for the signal handling is this:


static void setupUncaughtSignals()
{
    // Setup the handler function.
    struct sigaction signalAction;
    signalAction.sa_sigaction = signalHandler;
    signalAction.sa_flags = SA_SIGINFO;
    sigemptyset(&signalAction.sa_mask);

    for (int i = 0; i < n_fatal_signals; i++) {
        sigaction(signals[i], &signalAction, NULL);
    }
}

void resetSignalHandler(void)
{
    //reset the singal hander to the default
    struct sigaction signalAction;
    signalAction.sa_sigaction = signalHandler;
    signalAction.sa_handler = SIG_DFL;
    sigemptyset(&signalAction.sa_mask);

    for (int i = 0; i < n_fatal_signals; i++) {
        sigaction(signals[i], &signalAction, NULL);
    }
}

void signalHandler(int signal, siginfo_t *info, void *context)
{

    //reset the signal handler to use the default signal handler
    resetSignalHandler();

    [Logging signalHandler:signal];

    // After the signal is handled, we raise the signal again to the default handler
    raise(signal);
}

+ (void)signalHandler:(NSInteger)signal
{
    // Get the stacktrace and split it out into an array by every new line
    NSArray *stackTrace = [GTMStackTrace() componentsSeparatedByString:@"\n"];

    NSString *signalDescription;
    switch (signal)
    {
        case SIGQUIT:
            signalDescription = @"SIGQUIT - quit";
            break;
        case SIGILL:
            signalDescription = @"SIGILL - illegal instruction";
            break;
        case SIGABRT:
            signalDescription = @"SIGABRT - abort()";
            break;
        case SIGFPE:
            signalDescription = @"SIGFPE - floating point exception";
            break;
        case SIGBUS:
            signalDescription = @"SIGBUS - bus error";
            break;
        case SIGSEGV:
            signalDescription = @"SIGSEGV - segmentation violation";
            break;
        case SIGSYS:
            signalDescription = @"SIGSYS - bad argument to system call";
            break;
        case SIGPIPE:
            signalDescription = @"SIGPIPE - write on a pipe with no one to read it";
            break;
        case SIGXCPU:
            signalDescription = @"SIGXCPU - exceeded CPU time limit";
            break;
        case SIGXFSZ:
            signalDescription = @"SIGXFSZ - exceeded file size limit";
            break;
        default:
            signalDescription = @"Unrecognized signal. Check SignalId in signal.h for description.";
            break;
    }

    NSMutableDictionary *info = [self standardLoggingInfo];
    info[@"MessageType"] = kMsgTypeCrashReport;
    info[@"CrashReportVersion"] = CRASH_REPORT_VERSION;
    info[@"SignalId"] = @(signal);
    info[@"SignalDescription"] = signalDescription;

    NSArray *exceptionArray = @[info,
                                @{@"StackTrace" : stackTrace ?: @""},
                                @{@"DebugStatements": _ReportLoggerGetRecentLogs() ?: @""},
                                ];

    [self pushLoggingData:exceptionArray toLogFile:kSignalsLog asynchronously:NO];

    // Save the checkpoints buffer to disk
    [self pushCheckPointsToFile];
}


Using that code, this is an example of what I get (using on device symbolication, some symbols are missing):

"SignalId": 11, "SignalDescription": "SIGSEGV - segmentation violation",
"StackTrace": [
"#0 MyApp 0X000000010012823C +[Logging signalHandler:]",
"#1 libsystem_platform.dylib 0X0000000180E6594C _sigtramp()",
"#2 CoreFoundation 0X00000001810ECF00 +[__NSArrayI __new:::]",
"#3 CoreFoundation 0X00000001810ECF00 +[__NSArrayI __new:::]",
"#4 Foundation 0X0000000181B53C38 -[NSDictionary encodeWithCoder:]",
"#5 Foundation 0X0000000181AED900 -[NSXPCEncoder _encodeObject:]",
"#6 Foundation 0X0000000181AEDDE8 <redacted>()",
"#7 Foundation 0X0000000181AEDAA8 -[NSXPCEncoder encodeInvocation:]",
"#8 Foundation 0X0000000181AED900 -[NSXPCEncoder _encodeObject:]",
"#9 Foundation 0X0000000181CC3D34 -[NSXPCConnection _sendInvocation:withProxy:remoteInterface:withErrorHandler:timeout:userInfo:]",
"#10 CoreFoundation 0X000000018120AAA4 <redacted>()",
"#11 CoreFoundation 0X0000000181108D1C _CF_forwarding_prep_0()",
"#12 CFNetwork 0X0000000181839AD8 <redacted>()",
"#13 libdispatch.dylib 0X0000000180C514BC <redacted>()",
"#14 libdispatch.dylib 0X0000000180C5147C <redacted>()",
"#15 libdispatch.dylib 0X0000000180C5D4C0 <redacted>()",
"#16 libdispatch.dylib 0X0000000180C54F80 <redacted>()",
"#17 libdispatch.dylib 0X0000000180C5F390 <redacted>()",
"#18 libdispatch.dylib 0X0000000180C5F0B0 <redacted>()",
"#19 libsystem_pthread.dylib 0X0000000180E69470 _pthread_wqthread()",
"#20 libsystem_pthread.dylib 0X0000000180E69020 start_wqthread()"
]


Another:

"ExceptionReason": "*** -[NSXPCEncoder _checkObject:]: This coder only encodes objects that adopt NSSecureCoding (object is of class '__NSMallocBlock__')."
"StackTrace": [
"#0 CoreFoundation 0X248C010B <redacted>()",
"#1 libobjc.A.dylib 0X24066E17 objc_exception_throw()",
"#2 CoreFoundation 0X248C0051 -[NSException initWithCoder:]",
"#3 Foundation 0X250016C5 -[NSXPCEncoder _checkObject:]",
"#4 Foundation 0X25063369 -[NSXPCEncoder _encodeArrayOfObjects:forKey:]",
"#5 Foundation 0X250606CD -[NSDictionary encodeWithCoder:]",
"#6 Foundation 0X25001C37 -[NSXPCEncoder _encodeObject:]",
"#7 Foundation 0X2500206B <redacted>()",
"#8 Foundation 0X25001DC7 -[NSXPCEncoder encodeInvocation:]",
"#9 Foundation 0X25001C37 -[NSXPCEncoder _encodeObject:]",
"#10 Foundation 0X251ABED7 -[NSXPCConnection _sendInvocation:withProxy:remoteInterface:withErrorHandler:timeout:userInfo:]",
"#11 Foundation 0X251AB627 -[NSXPCConnection _sendInvocation:withProxy:remoteInterface:withErrorHandler:]",
"#12 Foundation 0X2501124F -[_NSXPCDistantObjectWithError forwardInvocation:]",
"#13 CoreFoundation 0X248C33FB <redacted>()",
"#14 CoreFoundation 0X247F3C08 -[NSKVONotifying__UIPopoverLayoutInfo setContentInset:]",
"#15 CFNetwork 0X24E2ACED -[NSURLStorage_CacheClient addCachedResponseWithDictionary:key:]",
"#16 CFNetwork 0X24DF6AAF <redacted>()",
"#17 libdispatch.dylib 0X24439B5B <redacted>()",
"#18 libdispatch.dylib 0X2444675B <redacted>()",
"#19 libdispatch.dylib 0X24445D99 <redacted>()",
"#20 libdispatch.dylib 0X24448495 <redacted>()",
"#21 libdispatch.dylib 0X24448305 <redacted>()",
"#22 libsystem_pthread.dylib 0X245F7B29 _pthread_wqthread()",
"#23 libsystem_pthread.dylib 0X245F7718 start_wqthread()"
]

In the application, I do have an uncaught exception handler and signal handler to generate crash reports and send them to Splunk.

Can you reproduce the problem with that code disabled? If so, please do, then post the crash you get then.

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

WWDC runs Mon, 13 Jun through to Fri, 17 Jun. During that time all of DTS will be at the conference, helping folks out face-to-face. http://developer.apple.com/wwdc/

I was reading through https://mikeash.com/pyblog/friday-qa-2011-04-01-signal-handling.html and I think based on the information inside, we're not appropriately handling the signals; I think that is why the crash report is misleading and pointing to the main thread rather than the CFNetwork dispatch thread. Unfortunately, I've not been able to reproduce this issue in-house and have only received reports from production deployments. The next step for me I guess is to replace the faulty signal/exception handling code with something more accurate and robust.

I'm getting similar crash thousands of times that collect from our users in a few days. And I also have a custom NSURLProtocol.

Here are three of these crash logs:

http://bugly.qq.com/share/DvIVna

http://bugly.qq.com/share/hA7vYx

http://bugly.qq.com/share/Rv4iEj


It makes me suffer a lot, can somebody help ? Thank you

To look deeper into this I need to see an Apple crash log of the problem.

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

WWDC runs Mon, 13 Jun through to Fri, 17 Jun. During that time all of DTS will be at the conference, helping folks out face-to-face. http://developer.apple.com/wwdc/

Hi eskimo :

Thank you for your reply!

My question may be very stupid, but why an Apple crash log is Absolute needed, does the three crash logs I provided above missing something that only contained in an Apple crash log?

Yes, the three crash logs above are generated by a third party companie, but they have the crash thread's stack info and all the other thead's. "出错线程" represent "crash thread", and "其他线程" represnet "other threads". Because of these pages are chinese, I afraid you should missed these contents. Or you didn't missed it at all.

Here, I provide you more crash logs that contains more info then the three above. You can find the thread state of the crash thread in the new logs, please have a look at it, and see dose it useful. Just click "其他信息" and the "更多信息" to find it.

Here are the new crash logs: http://pan.baidu.com/s/1slGdPQd. It's save as webarchive.

And also I will try to find an Apple Crash log of the problem. Because I can't reproduce the problem, it may be difficult to get one, but I'll try my best.

Thank You

My question may be very stupid, but why an Apple crash log is Absolute needed …

Because:

  1. the first thing I do is feed the crash log into an automated tool that tells me a bunch of info about the crash, which won’t work unless the crash log is in the standard format

  2. in my experience third-party crash logs leave out (or corrupt) a whole bunch of info that is necessary to debug the really tricky problems

With regards point 2, a perfect example of this is the crash log that started this thread; it pointed to thread A when the crash is actually happening in thread B. In this case it was easy to spot that discrepancy but if that hadn’t been the case then we’d have wasted a bunch of time looking at the wrong thread.

I have extensive experience writing crash reporters and I can tell you in no uncertain terms that writing a crash reporter is:

  • really hard to do for any given release of the OS

  • impossible to do in a binary compatible fashion (sooner or later the OS will change in such a way that breaks your crash reporter)

  • especially difficult to do if you have to do it in process, as is the case on iOS-based platforms

I understand why folks use third-party crash reporters, and that’s their decision to make, but in my opinion they are making the wrong trade off.

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

WWDC runs Mon, 13 Jun through to Fri, 17 Jun. During that time all of DTS will be at the conference, helping folks out face-to-face. http://developer.apple.com/wwdc/

Thanks, I see now. And I will try to find an Apple Crash log of the problem. Please wait for it.


Here are the same problem I found on the Internet, they are reported one years ago, but there is still no solution for it. I think you can look at it for reference.



Best Regards

Hi eskimo:

You can download an Apple crash log of the problem from the link here:http://pan.baidu.com/s/1qY1n2I4

Our app has uses a third-party crash reporter named Bugly, Bugly's signal handler function rqd_signal_handler recevice a signal in Thread 19, and we can't see the code inside rqd_signal_handler. In the above I have provied a lot of Bugly's log.


Thread 19 name:
Thread 19:
0   libsystem_kernel.dylib         0x00000001824577ec __kill + 8
1   CTRIP_WIRELESS                 0x00000001002cc83c rqd_signal_handler(int, __siginfo*, void*) + 1368
2   libsystem_platform.dylib       0x000000018251d94c _sigtramp + 68 (sigtramp.c:255)
3   CoreFoundation                 0x00000001827a4f00 +[__NSArrayI __new:::] + 100 (NSCollectionAux.h:37)
4   CoreFoundation                 0x00000001827a4f00 +[__NSArrayI __new:::] + 100 (NSCollectionAux.h:37)
5   Foundation                     0x000000018320bc38 -[NSDictionary(NSDictionary) encodeWithCoder:] + 908 (NSDictionary.m:108)
6   Foundation                     0x00000001831a5900 -[NSXPCEncoder _encodeObject:] + 684 (NSXPCCoder.m:411)
7   Foundation                     0x00000001831a5de8 encodeInvocationArguments + 320 (NSDOSerialization.m:1186)
8   Foundation                     0x00000001831a5aa8 -[NSXPCEncoder encodeInvocation:] + 368 (NSXPCCoder.m:300)
9   Foundation                     0x00000001831a5900 -[NSXPCEncoder _encodeObject:] + 684 (NSXPCCoder.m:411)
10  Foundation                     0x000000018337bd34 -[NSXPCConnection _sendInvocation:withProxy:remoteInterface:withErrorHandler:timeout:userInfo:] + 2328 (NSXPCConnection.m:918)
11  CoreFoundation                 0x00000001828c2aa4 ___forwarding___ + 408 (NSForwarding.m:3162)
12  CoreFoundation                 0x00000001827c0d1c _CF_forwarding_prep_0 + 92 (NSForwarding.s:780)
13  CFNetwork                     0x0000000182ef1ad8 ___ZN12__CFURLCache23CreateAndStoreCacheNodeEP16__CFURLCacheNodePK20_CFCachedURLResponsePK10__CFStringPK13_CFURLRequestPKvbRb_block_invoke + 1964 (CFURLCache.mm:3464)
14  libdispatch.dylib             0x00000001823094bc _dispatch_call_block_and_release + 24 (init.c:760)
15  libdispatch.dylib             0x000000018230947c _dispatch_client_callout + 16 (object.m:506)
16  libdispatch.dylib             0x00000001823154c0 _dispatch_queue_drain + 864 (inline_internal.h:1063)
17  libdispatch.dylib             0x000000018230cf80 _dispatch_queue_invoke + 464 (queue.c:3644)
18  libdispatch.dylib             0x0000000182317390 _dispatch_root_queue_drain + 728 (inline_internal.h:1043)
19  libdispatch.dylib             0x00000001823170b0 _dispatch_worker_thread3 + 112 (queue.c:4249)
20  libsystem_pthread.dylib       0x0000000182521470 _pthread_wqthread + 1092 (pthread.c:1990)
21  libsystem_pthread.dylib       0x0000000182521020 start_wqthread + 4 (pthread_asm.s:190)


And like madsolar8582's example, this crash log is misleading and pointing to the main thread rather the Thread 19


Exception Type:  EXC_CRASH (SIGSEGV)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Exception Note:  EXC_CORPSE_NOTIFY
Triggered by Thread:  0

Thread 0 name:
Thread 0 Crashed:
0   ImageIO                       0x0000000184386ac4 DGifDecompressLine + 436 (dgif_lib.c:962)
1   ImageIO                       0x00000001843869d8 DGifDecompressLine + 200 (dgif_lib.c:851)
2   ImageIO                       0x00000001843868c4 _cg_DGifGetLine + 104 (dgif_lib.c:491)
3   ImageIO                       0x00000001843865d0 decodeGIFFrame + 556 (imageGif.c:571)
4   ImageIO                       0x0000000184385c40 copyImageBlockSetGIF + 2232 (imageGif.c:4070)
5   ImageIO                       0x0000000184343368 ImageProviderCopyImageBlockSetCallback + 856 (CGImagePlus.c:1648)
6   QuartzCore                     0x00000001853d2f14 CA::Render::create_image(CGImage*, CGColorSpace*, unsigned int) + 988 (render-image-cache.cpp:955)
7   QuartzCore                     0x00000001853d2030 CA::Render::copy_image(CGImage*, CGColorSpace*, unsigned int, double) + 400 (render-image-cache.cpp:1290)
8   QuartzCore                     0x00000001853d2b24 CA::Render::prepare_image(CGImage*, CGColorSpace*, unsigned int, double) + 12 (render-image-cache.cpp:1372)
9   QuartzCore                     0x00000001853a599c CA::Layer::prepare_commit(CA::Transaction*) + 292 (CALayer.mm:2565)
10  QuartzCore                     0x00000001853a4b30 CA::Context::commit_transaction(CA::Transaction*) + 264 (CAContextInternal.mm:1633)
11  QuartzCore                     0x00000001853a486c CA::Transaction::commit() + 512 (CATransactionInternal.mm:417)
12  QuartzCore                     0x00000001853f824c CA::Display::DisplayLink::dispatch_items(unsigned long long, unsigned long long, unsigned long long) + 592 (CADisplay.mm:1214)
13  IOKit                         0x0000000182b35e54 IODispatchCalloutFromCFMessage + 372 (IOKitLib.c:1190)
14  CoreFoundation                 0x000000018285d0e0 __CFMachPortPerform + 180 (CFMachPort.c:553)
15  CoreFoundation                 0x000000018287585c __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE1_PERFORM_FUNCTION__ + 56 (CFRunLoop.c:1777)
16  CoreFoundation                 0x0000000182874f94 __CFRunLoopDoSource1 + 436 (CFRunLoop.c:1896)
17  CoreFoundation                 0x0000000182872cec __CFRunLoopRun + 1800 (CFRunLoop.c:2740)
18  CoreFoundation                 0x000000018279cd10 CFRunLoopRunSpecific + 384 (CFRunLoop.c:2814)
19  GraphicsServices               0x0000000184084088 GSEventRunModal + 180 (GSEvent.c:2245)
20  UIKit                         0x0000000187a71f70 UIApplicationMain + 204 (UIApplication.m:3772)
21  CTRIP_WIRELESS                 0x0000000100095840 main (main.m:19)
22  libdyld.dylib                 0x000000018233a8b8 start + 4 (start_glue.s:78)


Does this crash log useful? Do you have any idea about why the Thread 19 would crashed?

Hi eskimo:

Can you help me to take a look at the post above I replyed?

Thank you very much!

Can you help me to take a look at the post above I replyed?

Sorry I didn’t respond sooner; you posted while I was out of the office.

Again, this is a non-standard crash report and thus there’s limits to what I can do with it. Specifically, it’s not amenable to the automated analysis that I mentioned in point 1 of my 11 Jul post.

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

lao tie, zhao dao jie jue ban fa le ma? wo ye yu dao le zhe ge wen ti

Crash in com.apple.CFNetwork.addPersistCacheToStorageDaemon
 
 
Q