Crashes after exceeding limit of 150 wakeups per second over 300 seconds

Question

Created Jun ’24

Replies 2

Boosts 0

Views 1.1k

Participants 2

my macOS process is crashing when i keep my mac sleep for about 1 hour. Under Crash Reports, Console app shows two type of file.

.diag type
.ips type

ips file doesn't shows exact line of crash as you can see below sample.

 Thread 67 Crashed:
0   libsystem_kernel.dylib                   0x19a4aea60 __pthread_kill + 8
1   libsystem_pthread.dylib                  0x19a4e6c20 pthread_kill + 288
2   libsystem_c.dylib                        0x19a3f3a30 abort + 180
3   libsystem_malloc.dylib                   0x19a303dc4 malloc_vreport + 896
4   libsystem_malloc.dylib                   0x19a307430 malloc_report + 64
5   libsystem_malloc.dylib                   0x19a321494 find_zone_and_free + 528
6   Firewall.so                              0x103c8a744 TunnelSendQueue::ResumeSend() + 460

resource.diag file sgowing warning about exceeding limit of 150 wakeups per second over 300 seconds. attached here.

reasource_consumptions_W36RNW09G.wakeups_resource_diag.txt

Date/Time:        2024-05-29 12:06:38.356 +0530
End time:         2024-05-29 12:11:34.196 +0530
OS Version:       macOS 14.2.1 (Build 23C71)
Architecture:     arm64e
Report Version:   44
Incident Identifier: 02DC8332-F921-4AED-8F08-53F1F4DE3FA5

Data Source:      Microstackshots
Shared Cache:     F9DDD844-7F3F-34BD-BE29-F0C72D5E5449 slid base address 0x18d5d8000, slide 0xd5d8000

Command:          scqd
Path:             /usr/local/org/Scq/bin/scqd
Resource Coalition ID: 725
Architecture:     arm64
Parent:           UNKNOWN [1]
PID:              34079

Event:            wakeups
Action taken:     none
Wakeups:          45001 wakeups over the last 296 seconds (152 wakeups per second average), exceeding limit of 150 wakeups per second over 300 seconds
Wakeups limit:    45000
Limit duration:   300s
Wakeups caused:   45001
Wakeups duration: 296s
Duration:         295.84s
Duration Sampled: 80.48s
Steps:            18

Hardware model:   MacBookPro18,1
Active cpus:      10
HW page size:     16384
VM page size:     16384

Fan speed:        0 rpm
Advisory levels:  Battery -> 2, User -> 2, ThermalPressure -> 0, Combined -> 2
Free disk space:  326.66 GB/460.43 GB, low space threshold 3072 MB
Vnodes Available: 71.89% (180281/250764)

Preferred User Language: en-IN
Country Code:     IN
OS Cryptex File Extents: 1

Heaviest stack for the target process:
  18  thread_start + 8 (libsystem_pthread.dylib + 7740) [0x18da10e3c]
  18  _pthread_start + 136 (libsystem_pthread.dylib + 28724) [0x18da16034]
  10  void* std::__1::__thread_proxy>, Mowgli::EventCore::ThreadPool::AddPermanentThread(Mowgli::EventCore::ThreadPool::PermanentThreadId&, std::__1::function const&, std::__1::function const&)::$_1>>(void*) + 84 (libmowglicore.dylib + 6842980) [0x105bbea64]
  10  void std::__1::__thread_execute>, Mowgli::EventCore::ThreadPool::AddPermanentThread(Mowgli::EventCore::ThreadPool::PermanentThreadId&, std::__1::function const&, std::__1::function const&)::$_1>(std::__1::tuple>, Mowgli::EventCore::ThreadPool::AddPermanentThread(Mowgli::EventCore::ThreadPool::PermanentThreadId&, std::__1::function const&, std::__1::function const&)::$_1>&, std::__1::__tuple_indices<>) + 32 (libmowglicore.dylib + 6844020) [0x105bbee74]
  10  decltype(static_cast const&, std::__1::function const&)::$_1>(fp)()) std::__1::__invoke const&, std::__1::function const&)::$_1>(Mowgli::EventCore::ThreadPool::AddPermanentThread(Mowgli::EventCore::ThreadPool::PermanentThreadId&, std::__1::function const&, std::__1::function const&)::$_1&&) + 24 (libmowglicore.dylib + 6844056) [0x105bbee98]
  10  Mowgli::EventCore::ThreadPool::AddPermanentThread(Mowgli::EventCore::ThreadPool::PermanentThreadId&, std::__1::function const&, std::__1::function const&)::$_1::operator()() const + 24 (libmowglicore.dylib + 6844152) [0x105bbeef8]
  10  std::__1::function::operator()() const + 24 (libmowglicore.dylib + 5535108) [0x105a7f584]
  10  std::__1::__function::__value_func::operator()() const + 60 (libmowglicore.dylib + 5558896) [0x105a85270]
  7   std::__1::__function::__func const&)::$_2, std::__1::allocator const&)::$_2>, void ()>::operator()() + 28 (libmowglicore.dylib + 6585220) [0x105b7fb84]
  7   std::__1::__function::__alloc_func const&)::$_2, std::__1::allocator const&)::$_2>, void ()>::operator()() + 28 (libmowglicore.dylib + 6590436) [0x105b80fe4]
  7   void std::__1::__invoke_void_return_wrapper::__call const&)::$_2&>(Mowgli::EventCore::EventScheduler::Start(Mowgli::AutoRefPtr const&)::$_2&) + 28 (libmowglicore.dylib + 6590476) [0x105b8100c]
  7   decltype(static_cast const&)::$_2&>(fp)()) std::__1::__invoke const&)::$_2&>(Mowgli::EventCore::EventScheduler::Start(Mowgli::AutoRefPtr const&)::$_2&) + 24 (libmowglicore.dylib + 6590548) [0x105b81054]
  7   Mowgli::EventCore::EventScheduler::Start(Mowgli::AutoRefPtr const&)::$_2::operator()() const + 32 (libmowglicore.dylib + 6590612) [0x105b81094]
  5   Mowgli::EventCore::EventScheduler::RunRead(unsigned int) + 260 (libmowglicore.dylib + 6515920) [0x105b6ecd0]
  5   Mowgli::EventCore::EventScheduler::RunOneRead(unsigned int) + 60 (libmowglicore.dylib + 6516980) [0x105b6f0f4]
  3   Mowgli::EventCore::FDEventScheduler::RunOneRead(unsigned int) + 104 (libmowglicore.dylib + 6649860) [0x105b8f804]
  3   kevent + 8 (libsystem_kernel.dylib + 31168) [0x18d9db9c0]


Powerstats for:   scqd [34079]
UUID:             9F403F74-FF43-322D-8495-7BD2A6FAFC32
Path:             /usr/local/org/Scq/bin/scqd
Resource Coalition ID: 725
Architecture:     arm64
Parent:           UNKNOWN [1]
UID:              0
Footprint:        22.23 MB
Start time:       2024-05-29 12:10:12.355 +0530
End time:         2024-05-29 12:11:32.839 +0530
Num samples:      18 (100%)
Primary state:    13 samples Non-Frontmost App, Non-Suppressed, Kernel mode, Effective Thread QoS Default, Requested Thread QoS Default, Override Thread QoS Unspecified
User Activity:    0 samples Idle, 18 samples Active
Power Source:     18 samples on Battery, 0 samples on AC
  18  thread_start + 8 (libsystem_pthread.dylib + 7740) [0x18da10e3c]
    18  _pthread_start + 136 (libsystem_pthread.dylib + 28724) [0x18da16034]
      10  void* std::__1::__thread_proxy>, Mowgli::EventCore::ThreadPool::AddPermanentThread(Mowgli::EventCore::ThreadPool::PermanentThreadId&, std::__1::function const&, std::__1::function const&)::$_1>>(void*) + 84 (libmowglicore.dylib + 6842980) [0x105bbea64]
        10  void std::__1::__thread_execute>, Mowgli::EventCore::ThreadPool::AddPermanentThread(Mowgli::EventCore::ThreadPool::PermanentThreadId&, std::__1::function const&, std::__1::function const&)::$_1>(std::__1::tuple>, Mowgli::EventCore::ThreadPool::AddPermanentThread(Mowgli::EventCore::ThreadPool::PermanentThreadId&, std::__1::function const&, std::__1::function const&)::$_1>&, std::__1::__tuple_indices<>) + 32 (libmowglicore.dylib + 6844020) [0x105bbee74]
          10  decltype(static_cast const&, std::__1::function const&)::$_1>(fp)()) std::__1::__invoke const&, std::__1::function const&)::$_1>(Mowgli::EventCore::ThreadPool::AddPermanentThread(Mowgli::EventCore::ThreadPool::PermanentThreadId&, std::__1::function const&, std::__1::function const&)::$_1&&) + 24 (libmowglicore.dylib + 6844056) [0x105bbee98]
            10  Mowgli::EventCore::ThreadPool::AddPermanentThread(Mowgli::EventCore::ThreadPool::PermanentThreadId&, std::__1::function const&, std::__1::function const&)::$_1::operator()() const + 24 (libmowglicore.dylib + 6844152) [0x105bbeef8]
              10  std::__1::function::operator()() const + 24 (libmowglicore.dylib + 5535108) [0x105a7f584]
                10  std::__1::__function::__value_func::operator()() const + 60 (libmowglicore.dylib + 5558896) [0x105a85270]
                  7   std::__1::__function::__func const&)::$_2, std::__1::allocator const&)::$_2>, void ()>::operator()() + 28 (libmowglicore.dylib + 6585220) [0x105b7fb84]
                    7   std::__1::__function::__alloc_func const&)::$_2, std::__1::allocator const&)::$_2>, void ()>::operator()() + 28 (libmowglicore.dylib + 6590436) [0x105b80fe4]
                      7   void std::__1::__invoke_void_return_wrapper::__call const&)::$_2&>(Mowgli::EventCore::EventScheduler::Start(Mowgli::AutoRefPtr const&)::$_2&) + 28 (libmowglicore.dylib + 6590476) [0x105b8100c]
                        7   decltype(static_cast const&)::$_2&>(fp)()) std::__1::__invoke const&)::$_2&>(Mowgli::EventCore::EventScheduler::Start(Mowgli::AutoRefPtr const&)::$_2&) + 24 (libmowglicore.dylib + 6590548) [0x105b81054]
                          7   Mowgli::EventCore::EventScheduler::Start(Mowgli::AutoRefPtr const&)::$_2::operator()() const + 32 (libmowglicore.dylib + 6590612) [0x105b81094]
                            5   Mowgli::EventCore::EventScheduler::RunRead(unsigned int) + 260 (libmowglicore.dylib + 6515920) [0x105b6ecd0]
                              5   Mowgli::EventCore::EventScheduler::RunOneRead(unsigned int) + 60 (libmowglicore.dylib + 6516980) [0x105b6f0f4]
                                3   Mowgli::EventCore::FDEventScheduler::RunOneRead(unsigned int) + 104 (libmowglicore.dylib + 6649860) [0x105b8f804]
                                  3   kevent + 8 (libsystem_kernel.dylib + 31168) [0x18d9db9c0]
                                1   Mowgli::EventCore::FDEventScheduler::RunOneRead(unsigned int) + 1072 (libmowglicore.dylib + 6650828) [0x105b8fbcc]
                                  1   Mowgli::EventCore::BaseTimeoutEventScheduler::RunOne(unsigned int) + 56 (libmowglicore.dylib + 6651836) [0x105b8ffbc]
                                    1   Mowgli::EventCore::ThreadSafePriorityQueue>>, Mowgli::AutoRefPtr>, std::__1::greater>>, Mowgli::AutoRefPtr>>>::Empty() const + 52 (libmowglicore.dylib + 6540132) [0x105b74b64]
                                      1   
                                1   Mowgli::EventCore::FDEventScheduler::RunOneRead(unsigned int) + 36 (libmowglicore.dylib + 6649792) [0x105b8f7c0]
                                  1   Mowgli::EventCore::BaseTimeoutEventScheduler::Stopped() const + 40 (libmowglicore.dylib + 6650904) [0x105b8fc18]
                                    1   std::__1::shared_lock::shared_lock(std::__1::shared_mutex&) + 36 (libmowglicore.dylib + 6675880) [0x105b95da8]
                                      1   std::__1::shared_lock::shared_lock(std::__1::shared_mutex&) + 60 (libmowglicore.dylib + 6676000) [0x105b95e20]
                                        1   std::__1::shared_mutex::lock_shared() + 24 (libmowglicore.dylib + 6676060) [0x105b95e5c]
                                          1   std::__1::__shared_mutex_base::lock_shared() + 124 (libc++.1.dylib + 106632) [0x18d944088]
                                            1   std::__1::mutex::unlock() + 16 (libc++.1.dylib + 90956) [0x18d94034c]
                                              1   pthread_mutex_unlock + 72 (libsystem_pthread.dylib + 7060) [0x18da10b94]
                                                1   
                            1   Mowgli::EventCore::EventScheduler::RunRead(unsigned int) + 396 (libmowglicore.dylib + 6516056) [0x105b6ed58]
                              1   Mowgli::EventCore::BaseTimeoutEventScheduler::IsIdle() const + 32 (libmowglicore.dylib + 6516424) [0x105b6eec8]
                                1   std::__1::__atomic_base::operator int() const + 28 (libmowglicore.dylib + 6533264) [0x105b73090]
                                  1   int std::__1::__cxx_atomic_load(std::__1::__cxx_atomic_base_impl const*, std::__1::memory_order) + 108 (libmowglicore.dylib + 6533712) [0x105b73250]
                                    1   
                            1   Mowgli::EventCore::EventScheduler::RunRead(unsigned int) + 308 (libmowglicore.dylib + 6515968) [0x105b6ed00]
                              1   Mowgli::EventCore::BaseTimeoutEventScheduler::RunOne(unsigned int) + 48 (libmowglicore.dylib + 6518056) [0x105b6f528]
                                1   Mowgli::AutoRefPtr::~AutoRefPtr() + 0 (libmowglicore.dylib + 1819280) [0x1056f4290]
                                  1   
                  3   std::__1::__function::__func const&)::$_4, std::__1::allocator const&)::$_4>, void ()>::operator()() + 28 (libmowglicore.dylib + 6599464) [0x105b83328]
                    3   std::__1::__function::__alloc_func const&)::$_4, std::__1::allocator const&)::$_4>, void ()>::operator()() + 28 (libmowglicore.dylib + 6604680) [0x105b84788]
                      3   void std::__1::__invoke_void_return_wrapper::__call const&)::$_4&>(Mowgli::EventCore::EventScheduler::Start(Mowgli::AutoRefPtr const&)::$_4&) + 28 (libmowglicore.dylib + 6604720) [0x105b847b0]
                        3   decltype(static_cast const&)::$_4&>(fp)()) std::__1::__invoke const&)::$_4&>(Mowgli::EventCore::EventScheduler::Start(Mowgli::AutoRefPtr const&)::$_4&) + 24 (libmowglicore.dylib + 6604792) [0x105b847f8]
                          3   Mowgli::EventCore::EventScheduler::Start(Mowgli::AutoRefPtr const&)::$_4::operator()() const + 32 (libmowglicore.dylib + 6604856) [0x105b84838]
                            3   Mowgli::EventCore::EventScheduler::RunWrite(unsigned int) + 192 (libmowglicore.dylib + 6519176) [0x105b6f988]
                              3   Mowgli::EventCore::EventScheduler::RunOneWrite(unsigned int) + 60 (libmowglicore.dylib + 6519348) [0x105b6fa34]
                                3   Mowgli::EventCore::FDEventScheduler::RunOneWrite(unsigned int) + 64 (libmowglicore.dylib + 6652760) [0x105b90358]
                                  3   kevent + 8 (libsystem_kernel.dylib + 31168) [0x18d9db9c0]
      6   void* std::__1::__thread_proxy>, Mowgli::Log::LogList::StartThreading()::$_1>>(void*) + 84 (libmowglicore.dylib + 4978248) [0x1059f7648]
        6   void std::__1::__thread_execute>, Mowgli::Log::LogList::StartThreading()::$_1>(std::__1::tuple>, Mowgli::Log::LogList::StartThreading()::$_1>&, std::__1::__tuple_indices<>) + 32 (libmowglicore.dylib + 4979144) [0x1059f79c8]
          6   decltype(static_cast(fp)()) std::__1::__invoke(Mowgli::Log::LogList::StartThreading()::$_1&&) + 24 (libmowglicore.dylib + 4979180) [0x1059f79ec]
            6   Mowgli::Log::LogList::StartThreading()::$_1::operator()() const + 28 (libmowglicore.dylib + 4979280) [0x1059f7a50]
              6   Mowgli::Log::LogList::LogTimer() + 100 (libmowglicore.dylib + 4967172) [0x1059f4b04]
                6   void std::__1::this_thread::sleep_for>(std::__1::chrono::duration> const&) + 172 (libmowglicore.dylib + 4961132) [0x1059f336c]
                  6   std::__1::this_thread::sleep_for(std::__1::chrono::duration> const&) + 84 (libc++.1.dylib + 137244) [0x18d94b81c]
                    4   nanosleep + 116 (libsystem_c.dylib + 53904) [0x18d8b8290]
                      4   clock_get_time + 120 (libsystem_kernel.dylib + 28672) [0x18d9db000]
                        4   mach_msg2_trap + 8 (libsystem_kernel.dylib + 6260) [0x18d9d5874]
                    2   __semwait_signal + 8 (libsystem_kernel.dylib + 20072) [0x18d9d8e68]
      1   void* std::__1::__thread_proxy>, Mowgli::EventCore::ThreadPool::Start()::$_0>>(void*) + 84 (libmowglicore.dylib + 6824596) [0x105bba294]
        1   void std::__1::__thread_execute>, Mowgli::EventCore::ThreadPool::Start()::$_0>(std::__1::tuple>, Mowgli::EventCore::ThreadPool::Start()::$_0>&, std::__1::__tuple_indices<>) + 32 (libmowglicore.dylib + 6826616) [0x105bbaa78]
          1   decltype(static_cast(fp)()) std::__1::__invoke(Mowgli::EventCore::ThreadPool::Start()::$_0&&) + 24 (libmowglicore.dylib + 6826716) [0x105bbaadc]
            1   Mowgli::EventCore::ThreadPool::Start()::$_0::operator()() const + 72 (libmowglicore.dylib + 6826860) [0x105bbab6c]
              1   Mowgli::EventCore::ThreadPool::ThreadWorker(unsigned int) + 116 (libmowglicore.dylib + 6803372) [0x105bb4fac]
                1   Mowgli::EventCore::ThreadPool::SafeCall(std::__1::function const&) + 36 (libmowglicore.dylib + 6804124) [0x105bb529c]
                  1   std::__1::function::operator()() const + 24 (libmowglicore.dylib + 5535108) [0x105a7f584]
                    1   std::__1::__function::__value_func::operator()() const + 60 (libmowglicore.dylib + 5558896) [0x105a85270]
                      1   std::__1::__function::__func::HandleDeferredCall(Mowgli::AutoRefPtr, unsigned int, int)::'lambda'(), std::__1::allocator::HandleDeferredCall(Mowgli::AutoRefPtr, unsigned int, int)::'lambda'()>, void ()>::operator()() + 28 (libmowglicore.dylib + 6571348) [0x105b7c554]
                        1   std::__1::__function::__alloc_func::HandleDeferredCall(Mowgli::AutoRefPtr, unsigned int, int)::'lambda'(), std::__1::allocator::HandleDeferredCall(Mowgli::AutoRefPtr, unsigned int, int)::'lambda'()>, void ()>::operator()() + 28 (libmowglicore.dylib + 6576468) [0x105b7d954]
                          1   void std::__1::__invoke_void_return_wrapper::__call::HandleDeferredCall(Mowgli::AutoRefPtr, unsigned int, int)::'lambda'()&>(Mowgli::EventCore::BaseTimeoutEventScheduler::HandleDeferredCall(Mowgli::AutoRefPtr, unsigned int, int)::'lambda'()&) + 28 (libmowglicore.dylib + 6576508) [0x105b7d97c]
                            1   decltype(static_cast::HandleDeferredCall(Mowgli::AutoRefPtr, unsigned int, int)::'lambda'()&>(fp)()) std::__1::__invoke::HandleDeferredCall(Mowgli::AutoRefPtr, unsigned int, int)::'lambda'()&>(Mowgli::EventCore::BaseTimeoutEventScheduler::HandleDeferredCall(Mowgli::AutoRefPtr, unsigned int, int)::'lambda'()&) + 24 (libmowglicore.dylib + 6576580) [0x105b7d9c4]
                              1   Mowgli::EventCore::BaseTimeoutEventScheduler::HandleDeferredCall(Mowgli::AutoRefPtr, unsigned int, int)::'lambda'()::operator()() const + 316 (libmowglicore.dylib + 6576928) [0x105b7db20]
                                1   Mowgli::EventCore::BaseEvent::OnEvent() + 36 (libmowglicore.dylib + 18944) [0x10553ca00]
                                  1   Mowgli::AutoRefPtr Mowgli::EventCore::AddNewGenericEvent(Mowgli::AutoRefPtr, Mowgli::AutoRefPtr, Mowgli::ExecutionContext::Interface_ExecuteAfterInterruption(Mowgli::IEventScheduler*, Mowgli::IThreadPool*, void (*)(void*), void*)::'lambda'())::'lambda'(Mowgli::EventCore::GenericEvent&)>(Mowgli::AutoRefPtr, void*, bool, unsigned int, Mowgli::ExecutionContext::Interface_ExecuteAfterInterruption(Mowgli::IEventScheduler*, Mowgli::IThreadPool*, void (*)(void*), void*)::'lambda'())::MyGenericEvent::OnEventImpl() + 36 (libmowglicore.dylib + 1820480) [0x1056f4740]
                                    1   void Mowgli::EventCore::ChainEvent(Mowgli::AutoRefPtr, Mowgli::AutoRefPtr, Mowgli::ExecutionContext::Interface_ExecuteAfterInterruption(Mowgli::IEventScheduler*, Mowgli::IThreadPool*, void (*)(void*), void*)::'lambda'())::'lambda'(Mowgli::EventCore::GenericEvent&)::operator()(Mowgli::EventCore::GenericEvent&) const + 52 (libmowglicore.dylib + 1820628) [0x1056f47d4]
                                      1   Mowgli::ExecutionContext::Interface_ExecuteAfterInterruption(Mowgli::IEventScheduler*, Mowgli::IThreadPool*, void (*)(void*), void*)::'lambda'()::operator()() const + 32 (libmowglicore.dylib + 1820780) [0x1056f486c]
                                        1   resumeTunnelReceive(void*) + 1428 (Firewall.so + 560436) [0x1044c4d34]
                                          1   Wireguard_IsConnected(WireguardConnectionPtr, Mowgli::IContext*) + 1776 (Firewall.so + 301616) [0x104485a30]
                                            1   Mowgli::Variant::Interface_SetUserDefined(Mowgli::IUserDefinedType*) + 60 (libmowglicore.dylib + 4653240) [0x1059a80b8]
                                              1   Mowgli::Value::Assign(Mowgli::AutoRefPtr&&) + 120 (libmowglicore.dylib + 4653416) [0x1059a8168]
                                                1   Mowgli::Value::Clear(bool) + 60 (libmowglicore.dylib + 4437388) [0x10597358c]
                                                  1   std::__1::basic_string, std::__1::allocator>::__is_long() const + 0 (libmowglicore.dylib + 4480820) [0x10597df34]
                                                    1   
      1   boost::(anonymous namespace)::thread_proxy(void*) + 180 (libboost_thread.dylib + 10552) [0x100eaa938]
        1   IThread::svc() + 48 (libscp.dylib + 891972) [0x1046edc44]
          1   ConnectivityChecker::onThread() + 4152 (libscp.dylib + 1851904) [0x1047d8200]
            1   ConnectivityChecker::checkConnectivity() + 4796 (libscp.dylib + 1890760) [0x1047e19c8]
              1   ConnectivityChecker::getReachablePrimaryServers(std::__1::list>&, bool, bool) + 768 (libscp.dylib + 1932756) [0x1047ebdd4]
                1   TcpConnect::start(unsigned short, std::__1::basic_string, std::__1::allocator> const&, SemaphoreBoost&) + 1264 (libscp.dylib + 1986756) [0x1047f90c4]
                  1   ConnectSession::connectToRemoteHost(char const*, int) + 1004 (libscp.dylib + 2121400) [0x104819eb8]
                    1   boost::asio::ip::basic_resolver::resolve(boost::asio::ip::tcp const&, std::__1::basic_string_view>, std::__1::basic_string_view>, boost::asio::ip::resolver_base::flags) + 364 (libscp.dylib + 2130424) [0x10481c1f8]
                      1   boost::asio::detail::resolver_service::resolve(std::__1::shared_ptr&, boost::asio::ip::basic_resolver_query const&, boost::system::error_code&) + 136 (libscp.dylib + 2294888) [0x104844468]
                        1   boost::asio::detail::socket_ops::getaddrinfo(char const*, char const*, addrinfo const&, addrinfo**, boost::system::error_code&) + 108 (libscp.dylib + 2295372) [0x10484464c]
                          1   getaddrinfo + 168 (libsystem_info.dylib + 21936) [0x18da4e5b0]
                            1   si_addrinfo + 1312 (libsystem_info.dylib + 23416) [0x18da4eb78]
                              1   search_addrinfo + 176 (libsystem_info.dylib + 51900) [0x18da55abc]
                                1   mdns_addrinfo + 360 (libsystem_info.dylib + 52340) [0x18da55c74]
                                  1   _mdns_search_ex + 668 (libsystem_info.dylib + 41720) [0x18da532f8]
                                    1   _mdns_query_start + 624 (libsystem_info.dylib + 44892) [0x18da53f5c]
                                      1   DNSServiceQueryRecordInternal + 716 (libsystem_dnssd.dylib + 16268) [0x19a299f8c]
                                        1   __close_nocancel + 8 (libsystem_kernel.dylib + 12472) [0x18d9d70b8]

  Binary Images:
           0x100828000 -                ???  scqd                                   <9F403F74-FF43-322D-8495-7BD2A6FAFC32>  /usr/local/org/Scq/bin/scqd
           0x100ea8000 -        0x100ebbfff  libboost_thread.dylib (0)              <B1F12E19-7DF5-3AE7-A656-3E70C2C52ED1>  /usr/local/org/fmp/lib64/libboost_thread.dylib
           0x10443c000 -        0x10450ffff  Firewall.so (0)                        <68BDCBB9-4B4A-340A-BE53-177D8FD996ED>  /usr/local/org/Scq/mowgli_files/mowgli_modules/Firewall.so
           0x104614000 -        0x104a6bfff  libscp.dylib (0)                       <83719E19-623F-354D-ADE6-C4348D08560E>  /usr/local/org/Scq/lib/libscp.dylib
           0x105538000 -        0x105d3bfff  libmowglicore.dylib (0)                <79694D53-A7E8-3D2B-800C-B8D76EE4AB47>  /usr/local/org/Scq/lib/libmowglicore.dylib
           0x18d8ab000 -        0x18d929ffb  libsystem_c.dylib (1583.60.2)          <AB8A0A0B-9A1E-3284-9243-5A0334468AE5>  /usr/lib/system/libsystem_c.dylib
           0x18d92a000 -        0x18d9b7ff7  libc++.1.dylib (1600.157)              <375254EB-5A65-35D0-B6B4-773E6763D289>  /usr/lib/libc++.1.dylib
           0x18d9d4000 -        0x18da0efff  libsystem_kernel.dylib (10002.61.3)    <CA94FC21-BC40-3B43-B65D-B87ECE9E1D48>  /usr/lib/system/libsystem_kernel.dylib
           0x18da0f000 -        0x18da1bff3  libsystem_pthread.dylib (519)          <A7D94C96-7B1F-3229-9BEA-048D037C3292>  /usr/lib/system/libsystem_pthread.dylib
           0x18da49000 -        0x18da75ffb  libsystem_info.dylib (583.0.1)         <0B3340BE-7CBE-356F-9ACE-7A062DC0554E>  /usr/lib/system/libsystem_info.dylib
           0x19a296000 -        0x19a29eff7  libsystem_dnssd.dylib (2200.60.25.0.4) <4FCBB463-F915-38A4-AB32-159AF74479E9>  /usr/lib/system/libsystem_dnssd.dylib

is this something macOS stopping app because of some resource consumptions?

Answered by DTS Engineer in 789922022

Splitting my answers up and starting with the crash here:

5 libsystem_malloc.dylib 0x19a321494 find_zone_and_free + 528

libmalloc is one of our opensource projects, so you can often find exactly what failed/printed by looking at the source itself. There is only one error message there, so find_zone_and_free in malloc.c printed this:

		malloc_report(flags,
				"*** error for object %p: pointer being freed was not allocated\n", ptr);

Which means there is some kind of memory corruption going on.

-Kevin Elliott
DTS Engineer, CoreOS/Hardware

Boost

Answer 1

DTS Engineer OP

Apple

Jun ’24

Accepted Answer

Splitting my answers up and starting with the crash here:

5 libsystem_malloc.dylib 0x19a321494 find_zone_and_free + 528

libmalloc is one of our opensource projects, so you can often find exactly what failed/printed by looking at the source itself. There is only one error message there, so find_zone_and_free in malloc.c printed this:

		malloc_report(flags,
				"*** error for object %p: pointer being freed was not allocated\n", ptr);

Which means there is some kind of memory corruption going on.

-Kevin Elliott
DTS Engineer, CoreOS/Hardware

1

Answer 2

DTS Engineer OP

Apple

Jun ’24

Next, on the "resource.diag" file, let me start with the immediate concern here:

is this something macOS stopping app because of some resource consumptions?

macOS (or any of our other systems) did not stop your app. As far as the user is concerned, this log didn't have any effect on your app. It's easy to overlook, but what this line means:

Action taken:     none

Is that the the system generated that log your looking at but took no other action. There are other situations which generate the same log type, but the log will always say "none" unless we did "something".

Moving to the details of the log itself, I'll start here:

 Wakeups:          45001 wakeups over the last 296 seconds (152 wakeups per second average), exceeding limit of 150 wakeups per second over 300 seconds

Quinn has a good write up of what "wakeup" means which is worth reviewing.

That leads to the spindump itself. Spindumps work by capturing a fixed number of thread stacks over a specific period of time. This is what's in the log here:

 Duration Sampled: 80.48s
 Steps:            18

That is, the kernel captured this thread stach 18 times over an ~80s time window. For display purposes, all of those samples are then "merged together" to created the nested stack you're looking at. You'll notice it starts with:

  18  thread_start + 8 (libsystem_pthread.dylib + 7740) [0x18da10e3c]
    18  _pthread_start + 136 (libsystem_pthread.dylib + 28724) [0x18da16034]

...meaning "all 18 tread stacks start with thread_start-> _pthread_start", because how a running pthread always starts. The nesting process continues from there, "breaking up" each of those 18 samples in to the different code paths each sample followed. Look at the next level, that shows:

      10  void* std::__1::__thread_proxy>, Mowgli::EventCore::ThreadPool::AddPermanentThread(Mowgli::EventCore::ThreadPool::PermanentThreadId&, std::__1::function const&, std::__1::function const&)::$_1>>(void*) + 84 (libmowglicore.dylib + 6842980) [0x105bbea64]

      6   void* std::__1::__thread_proxy>, Mowgli::Log::LogList::StartThreading()::$_1>>(void*) + 84 (libmowglicore.dylib + 4978248) [0x1059f7648]

      1   void* std::__1::__thread_proxy>, Mowgli::EventCore::ThreadPool::Start()::$_0>>(void*) + 84 (libmowglicore.dylib + 6824596) [0x105bba294]

      1   boost::(anonymous namespace)::thread_proxy(void*) + 180 (libboost_thread.dylib + 10552) [0x100eaa938]

In other words, you have 18 samples (10 + 6 + 1 + 1) but across 4 different code paths instead of 1.

Taking that down one more level, following "10" above, you have another "direct" nest chain:

      10  void* std::__1::__thread_proxy>, Mowgli::EventCore::ThreadPool::AddPermanentThread(Mowgli::EventCore::ThreadPool::PermanentThreadId&, std::__1::function const&, std::__1::function const&)::$_1>>(void*) + 84 (libmowglicore.dylib + 6842980) [0x105bbea64]
        10  void std::__1::__thread_execute>, Mowgli::EventCore::ThreadPool::AddPermanentThread(Mowgli::EventCore::ThreadPool::PermanentThreadId&, std::__1::function const&, std::__1::function const&)::$_1>(std::__1::tuple>, Mowgli::EventCore::ThreadPool::AddPermanentThread(Mowgli::EventCore::ThreadPool::PermanentThreadId&, std::__1::function const&, std::__1::function const&)::$_1>&, std::__1::__tuple_indices<>) + 32 (libmowglicore.dylib + 6844020) [0x105bbee74]
          10  decltype(static_cast const&, std::__1::function const&)::$_1>(fp)()) std::__1::__invoke const&, std::__1::function const&)::$_1>(Mowgli::EventCore::ThreadPool::AddPermanentThread(Mowgli::EventCore::ThreadPool::PermanentThreadId&, std::__1::function const&, std::__1::function const&)::$_1&&) + 24 (libmowglicore.dylib + 6844056) [0x105bbee98]
            10  Mowgli::EventCore::ThreadPool::AddPermanentThread(Mowgli::EventCore::ThreadPool::PermanentThreadId&, std::__1::function const&, std::__1::function const&)::$_1::operator()() const + 24 (libmowglicore.dylib + 6844152) [0x105bbeef8]
              10  std::__1::function::operator()() const + 24 (libmowglicore.dylib + 5535108) [0x105a7f584]
                10  std::__1::__function::__value_func::operator()() const + 60 (libmowglicore.dylib + 5558896) [0x105a85270]

Which then splits into:


                  7   std::__1::__function::__func const&)::$_2, std::__1::allocator const&)::$_2>, void ()>::operator()() + 28 (libmowglicore.dylib + 6585220) [0x105b7fb84]

                  3   std::__1::__function::__func const&)::$_4, std::__1::allocator const&)::$_4>, void ()>::operator()() + 28 (libmowglicore.dylib + 6599464) [0x105b83328]

...this time splitting up the 10 samples (7 + 3) into two code paths.

Whether or not this kind of activity represents a "problem" for your app depends a lot on what's actually going on. The total count (45001) does sound problematic, but if you divide out the rate (450001/296) you're only slightly over (~152) the limit and I've seen much worse cases. However, the really answer here is that it depends on what your app was actually "doing" that put it over this limit. Many of the apps that hit this are doing so ENTIRELY by accident. For example, I've seen case where that app was messing between threads, but the messaging was being done for every single transaction/transfer/modification instead of being "throttled" by wall time or any other constraint.

I've fallen into this trap myself. ~Twenty years ago I was working on an app that implemented it's own copy engine when a coworker "hinted" that something was seemed off about my overall copy performance. Once I shifted to only sending progress updates every 0.1s, copies were suddenly 5x faster.

Similarly, there may be space for optimization and tuning, even if the basic "job" you're trying to do will inherently involve frequent wakeups.

-Kevin Elliott
DTS Engineer, CoreOS/Hardware

1