In the message before yours I have posted full log.
Yep, that's what I based my earlier post on. Unfortunately, all I can really talk about here is what happened in "our" code. There's clearly plenty of other thread activity happening as well which is probably involved in creating the problem, but I can't really be any more specific than that.
The interesting stuff for me, that NSArray is handling by EAAccessory, it is part of EAAccessory framework, not my Array. So I try to understand what I can do with thread to affect Main Thread in that way that Object could be deleted in that moment when it was created.
Yes, but that array is holding the systems references to underlying object that EAAccessory manipulates. I'm not sure of what's happening that actually creates the crash, but that's my best guess from comparing the crash log and our code.
Also, to clarify, I do think our code has some "involvement" here, as that array shouldn't really be able to be put into that state. However, that doesn't mean you don't also have an issue here. Our fix for this kind of threading issue often ends up exposing other issue that were previously hidden by the current crash, assuming it doesn't simply result in a different crash*.
*As a more direct and dramatic example of this, my recommendation to any developer who's app manages to cause a kernel panic is:
-
File a bug immediately. The kernel should never panic, so all kernel panics are bugs.
-
Figure out what you're doing that's causing the panic and stop immediately.
The reason for #2 is quite simple. I said the kernel shouldn't panic, I didn't say it had to keep running your app. As far as the kernel is concerned terminating your process is a perfectly reasonable way to avoid panic'ing, but that "fix" won't really improve your apps user experience.
And usually it is happens in background. And btw, it is only visible on PROD, so when optimization is turned on.
FYI, you can change the compiler optimization level for debug builds. It defaults to "0" because optimizations like instruction reordering confuse the debugger*, but you can still follow the "flow" pretty well even at "5".
Also, if you haven't already, take a look at "Diagnosing memory, thread, and crash issues early" and do some testing with those tools, particularly the Thread Sanitizer. I can't promise they'll find anything (unfortunately, bug like this are hard...), but these issues are hard enough to investigate that a tool that finds even SOME problems can be a huge help.
Our QA has never observed the bug.
Yes, that's pretty common. The nature of threading failures is that they often require very specific circumstances to reproduce and, in most cases, you won't known what those are until after you've found the problem. However, two things you can look at here:
-
If you're in contact with any users experiencing this issue or have any other data about them, what makes them "different" than other user? Why makes them "special" compared to everyone else? One thing I particularly like to look at here is the crash timestamps, looking for things like how long the app was running, what time the crash occurred, and what timezone the crash was in (which gives an indication of "where" the users might be).
-
What is your QA process actually testing. For accessory apps in particular, I think there are two different "extremes" that testing should be focused on:
-
Stressing the accessory and app to artificially induce failure. For example, leaks or other issues are more likely at "transition" points, so a testing process that forces the app to connect/disconnect hundreds/thousands of times can be a lot more useful than simply testing normal app use.
-
Replicating real world usage in a way that gets you good data about what actually went wrong. The specifics of how this work depend on what you're accessory actually is but one crash from an app build that generates detailed activity logs or from a developers who's aware of your app internal implementation details can be FAR more useful than a standalone crash log.
__
Kevin Elliott
DTS Engineer, CoreOS/Hardware
