CSIdentityQueryExecute - possible results access after release?

Question

Created 4d

Replies 3

Boosts 0

Participants 2

We have a simple function that retrieves users list via CSIdentityQueryCreate() and CSIdentityQueryExecute().

However, sometimes we get crashes in "com.apple.opendirectory.odxpc.xpc" queue and address sanitizer reports that something inside opendirectory related framework tries to access data that was released:

==327==ERROR: AddressSanitizer: global-buffer-overflow on address 0x000166890010 at pc 0x000109b51f98 bp 0x00016b6f58d0 sp 0x00016b6f5070
READ of size 5 at 0x000166890010 thread T242
    #0 0x000109b51f94 in memcpy+0x294 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x85f94)
    #1 0x000191c08e64 in __CFStringCreateImmutableFunnel3+0x4f8 (CoreFoundation:arm64e+0x2e64)
    #2 0x000191d2494c in ____CFBinaryPlistCreateObjectFiltered_block_invoke+0xa4 (CoreFoundation:arm64e+0x11e94c)
    #3 0x000191c39fc8 in __CFBinaryPlistCreateObjectFiltered+0x698 (CoreFoundation:arm64e+0x33fc8)
    #4 0x000191c3aff0 in __CFBinaryPlistCreateObjectFiltered+0x16c0 (CoreFoundation:arm64e+0x34ff0)
    #5 0x000191c20b98 in __CFTryParseBinaryPlist+0xe8 (CoreFoundation:arm64e+0x1ab98)
    #6 0x000191d11e14 in _CFPropertyListCreateWithData+0x98 (CoreFoundation:arm64e+0x10be14)
    #7 0x000191c20a2c in CFPropertyListCreateWithData+0x38 (CoreFoundation:arm64e+0x1aa2c)
    #8 0x00019b3a49d0 in ___odxpc_create_connection_block_invoke+0x104 (CFOpenDirectory:arm64e+0x29d0)
    #9 0x00019189e82c in _xpc_connection_call_event_handler+0x6c (libxpc.dylib:arm64e+0xe82c)
    #10 0x00019189d128 in _xpc_connection_mach_event+0x4a8 (libxpc.dylib:arm64e+0xd128)
    #11 0x0001919f98a0 in _dispatch_client_callout4+0xc (libdispatch.dylib:arm64e+0x1b8a0)
    #12 0x0001919fc10c in _dispatch_mach_msg_invoke+0x1cc (libdispatch.dylib:arm64e+0x1e10c)
    #13 0x0001919e81b4 in _dispatch_lane_serial_drain+0x148 (libdispatch.dylib:arm64e+0xa1b4)
    #14 0x0001919fce70 in _dispatch_mach_invoke+0x1d4 (libdispatch.dylib:arm64e+0x1ee70)
    #15 0x0001919e81b4 in _dispatch_lane_serial_drain+0x148 (libdispatch.dylib:arm64e+0xa1b4)
    #16 0x0001919e8e28 in _dispatch_lane_invoke+0x180 (libdispatch.dylib:arm64e+0xae28)
    #17 0x0001919f3260 in _dispatch_root_queue_drain_deferred_wlh+0x120 (libdispatch.dylib:arm64e+0x15260)
    #18 0x0001919f2ae4 in _dispatch_workloop_worker_thread+0x218 (libdispatch.dylib:arm64e+0x14ae4)
    #19 0x000191b93e1c in _pthread_wqthread+0x120 (libsystem_pthread.dylib:arm64e+0x2e1c)
    #20 0x000191b92b70 in start_wqthread+0x4 (libsystem_pthread.dylib:arm64e+0x1b70)

0x000166890010 is located 10946 bytes after 789838-byte region [0x0001667cc800,0x00016688d54e)
freed by thread T240 here:
    #0 0x000109b09480 in free+0x7c (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x3d480)
    #1 0x000191c25ca8 in __CFDataDeallocate+0x5c (CoreFoundation:arm64e+0x1fca8)
    #2 0x000191d56940 in _CFRelease+0x124 (CoreFoundation:arm64e+0x150940)
    #3 0x00019f8e4540 in DSIdentity::~DSIdentity()+0x80 (OSServices:arm64e+0xe540)
    #4 0x00019f8e45ec in DSUserIdentity::~DSUserIdentity()+0xc (OSServices:arm64e+0xe5ec)
    #5 0x000191d56940 in _CFRelease+0x124 (CoreFoundation:arm64e+0x150940)
    #6 0x000191c21b28 in __CFBasicHashDrain+0x1a8 (CoreFoundation:arm64e+0x1bb28)
    #7 0x000191d56940 in _CFRelease+0x124 (CoreFoundation:arm64e+0x150940)
    #8 0x00019f8ec1bc in DSIdentityQuery::~DSIdentityQuery()+0x88 (OSServices:arm64e+0x161bc)
    #9 0x00019f8ec220 in DSIdentityQuery::~DSIdentityQuery()+0xc (OSServices:arm64e+0x16220)
    #10 0x000191d56940 in _CFRelease+0x124 (CoreFoundation:arm64e+0x150940)
    #11 0x000107eda344 in eka::posix::memory::CFReleaseDeleter<__CSIdentityQuery*>::operator()(__CSIdentityQuery*) const+0x3c (libxxx.dylib:arm64+0x9a344)
    #12 0x000107eda1c8  (libxxx.dylib:arm64+0x9a1c8)
    #13 0x000107eda010  (libxxx.dylib:arm64+0x9a010)
    #14 0x000107e43424  (libx.dylib:axxrm64+0x3424)
    #15 0x000107e41cc4 in my::helper::EnumerateUsers(std::__1::function<void (my::helper::UserInfo&&, bool&)> const&, unsigned int)+0xf7c (libxxx.dylib:arm64+0x1cc4)
...
previously allocated by thread T240 here:
    #0 0x000109b0938c in malloc+0x78 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x3d38c)
    #1 0x0001919b39d4 in _malloc_type_malloc_outlined+0x60 (libsystem_malloc.dylib:arm64e+0x1d9d4)
    #2 0x000191d11c34 in __CFDataInit+0x214 (CoreFoundation:arm64e+0x10bc34)
    #3 0x00019f8e3228 in DSIdentityRecord::addJPEGData(char const*, unsigned int, unsigned int)+0x28 (OSServices:arm64e+0xd228)
    #4 0x00019f8e627c in DSIdentity::CreateIdentityFromDSAttributeList(__CFAllocator const*, unsigned int, tDataBuffer*, unsigned int, long, bool, DSIdentityAuthority&, tDirStatus*)+0x1c8 (OSServices:arm64e+0x1027c)
    #5 0x00019f8ecb64 in DSIdentityQuery::consumeRecordData(unsigned int, tDataBuffer*, long)+0xb4 (OSServices:arm64e+0x16b64)
    #6 0x00019f900e2c in findRecordsWithNames+0xd0 (OSServices:arm64e+0x2ae2c)
    #7 0x00019f8ec49c in DSIdentityQuery::performQuery(__CFError**)+0x1c4 (OSServices:arm64e+0x1649c)
    #8 0x00019f8ec5c4 in DSIdentityQuery::execute(unsigned long, __CFError**)+0x40 (OSServices:arm64e+0x165c4)
    #9 0x00019f8e06f4 in CSIdentityQueryExecute+0x2c (OSServices:arm64e+0xa6f4)
    #10 0x000107e4144c in my::helper::EnumerateUsers(std::__1::function<void (my::helper::UserInfo&&, bool&)> const&, unsigned int)+0x704 (libxxx.dylib:arm64+0x144c)

EnumerateUsers is our function which basically does this (simplified, error checking skipped):

unique_cfref<CSIdentityQueryRef> query(CSIdentityQueryCreate(kCFAllocatorDefault, kCSIdentityClassUser, CSGetLocalIdentityAuthority()));

CSIdentityQueryExecute(query.get(), 0, &error);

unique_cfref<CFArrayRef> userIdentities(CSIdentityQueryCopyResults(query.get()));

auto userIdentitiesCount = CFArrayGetCount(userIdentities.get());
for (CFIndex i = 0; i < userIdentitiesCount; ++i)
{
    // do something including calling getpwuid()
}

unique_cfref is just a std::unique_ptr wrapper that calls CFRelease when we goes out of scope.

Looks like something in OpenDirectory tries to access released data (weak references and delayed processing?). The problem seems to occur on macOS 15.

Is there any bug/regression, should we file a feedback? Or maybe we do something wrong?

Boost

Answer 1

DTS Engineer OP

Apple

4d

unique_cfref is just a std::unique_ptr wrapper that calls CFRelease when we go out of scope.

Looks like something in OpenDirectory tries to access released data (weak references and delayed processing?). The problem seems to occur on macOS 15.

No, that's not what's going on, though I can see why you might have thought that. The key details here are this note in the middle:

0x000166890010 is located 10946 bytes after 789838-byte region

ASAN includes information about "nearby" allocations because there often is a relationship between allocations that are "close" to each other. However, the problem here is that 10946 bytes is 10KB, which isn't really all that "close". It's possible there is some connection, but it's more likely that this was just the closest block ASAN could relate, not that they were actually tied to each other.

Similarly, what the code is actually doing is converting raw bytes it received through XPC into plist data (Array/Dictionary/String/etc.), which your larger app wouldn't really have ever had access to.

Is there any bug/regression, should we file a feedback?

Yes, please file a bug on this and post the bug number back here. Please make sure you attach the actual crash logs you've gotten, not just your ASAN traces.

Having said that...

Or maybe we do something wrong?

...I can't rule out the possibility that this is a bug in your app. Have you looked for patterns in the crash logs you've seen?

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

0

Answer 2

goal81 OP

3d

I've looked through several crashes and the only "pattern" I've noticed is something like this:

Thread 30:: thread manager
0   libsystem_kernel.dylib        	       0x19534dbb0 semaphore_wait_trap + 8
1   libdispatch.dylib             	       0x1951d9960 _dispatch_sema4_wait + 28
2   libdispatch.dylib             	       0x1951d9f10 _dispatch_semaphore_wait_slow + 132
3   DirectoryService              	       0x1aabdb340 _continue_query + 88
4   DirectoryService              	       0x1aabdae50 dsGetRecordList + 484
5   OSServices                    	       0x1a30f7de4 findRecordsWithNames + 136
6   OSServices                    	       0x1a30e34a0 DSIdentityQuery::performQuery(__CFError**) + 456
7   OSServices                    	       0x1a30e35c8 DSIdentityQuery::execute(unsigned long, __CFError**) + 68
8   OSServices                    	       0x1a30d76f8 CSIdentityQueryExecute + 48
9   libxxx.dylib            	       0x1032e9450 my::helper::EnumerateUsers(std::__1::function<void (macos::user_info::UserInfo&&, bool&)> const&, unsigned int) + 1800
...

Thread 42 Crashed::  Dispatch queue: com.apple.opendirectory.odxpc.xpc
0   libsystem_kernel.dylib        	       0x195356388 __pthread_kill + 8
1   libsystem_pthread.dylib       	       0x19538f88c pthread_kill + 296
2   libsystem_c.dylib             	       0x195298a3c abort + 124
3   libclang_rt.asan_osx_dynamic.dylib	       0x1050bdaa0 __sanitizer::Abort() + 84
4   libclang_rt.asan_osx_dynamic.dylib	       0x1050bd1c0 __sanitizer::Die() + 108
5   libclang_rt.asan_osx_dynamic.dylib	       0x1050a116c __asan::ScopedInErrorReport::~ScopedInErrorReport() + 1172
6   libclang_rt.asan_osx_dynamic.dylib	       0x1050a03c8 __asan::ReportGenericError(unsigned long, unsigned long, unsigned long, unsigned long, bool, unsigned long, unsigned int, bool) + 1968
7   libclang_rt.asan_osx_dynamic.dylib	       0x1050e1fa4 wrap_memcpy + 676
8   CoreFoundation                	       0x195400f18 __CFStringCreateImmutableFunnel3 + 1276
9   CoreFoundation                	       0x19551ca00 ____CFBinaryPlistCreateObjectFiltered_block_invoke + 168
10  CoreFoundation                	       0x19543207c __CFBinaryPlistCreateObjectFiltered + 1692
11  CoreFoundation                	       0x1954330a4 __CFBinaryPlistCreateObjectFiltered + 5828
12  CoreFoundation                	       0x195418c4c __CFTryParseBinaryPlist + 236
13  CoreFoundation                	       0x195509ec8 _CFPropertyListCreateWithData + 156
14  CoreFoundation                	       0x195418ae0 CFPropertyListCreateWithData + 60
15  CFOpenDirectory               	       0x19eba19d4 ___odxpc_create_connection_block_invoke + 264
16  libxpc.dylib                  	       0x195096830 _xpc_connection_call_event_handler + 112
17  libxpc.dylib                  	       0x19509512c _xpc_connection_mach_event + 1196
18  libdispatch.dylib             	       0x1951f18a4 _dispatch_client_callout4 + 16
19  libdispatch.dylib             	       0x1951f4110 _dispatch_mach_msg_invoke + 464
20  libdispatch.dylib             	       0x1951e01b8 _dispatch_lane_serial_drain + 332
21  libdispatch.dylib             	       0x1951f4e74 _dispatch_mach_invoke + 472
22  libdispatch.dylib             	       0x1951e01b8 _dispatch_lane_serial_drain + 332
23  libdispatch.dylib             	       0x1951e0e2c _dispatch_lane_invoke + 388
24  libdispatch.dylib             	       0x1951eb264 _dispatch_root_queue_drain_deferred_wlh + 292
25  libdispatch.dylib             	       0x1951eaae8 _dispatch_workloop_worker_thread + 540
26  libsystem_pthread.dylib       	       0x19538be64 _pthread_wqthread + 292
27  libsystem_pthread.dylib       	       0x19538ab74 start_wqthread + 8

I.e. crash occurs on OD related queue while another thread fetches users data via CSIdentityQueryExecute(). It seems that the query is in progress because thread waits for semaphore.

Regarding the feedback I think we'll file a report soon.

0

Answer 3

DTS Engineer OP

Apple

3d

I.e. crash occurs on OD-related queue while another thread fetches users’ data via CSIdentityQueryExecute(). It seems that the query is in progress because the thread waits for a semaphore.

Yes, that's true. However, looking at our code, there's no way that could be disrupting the other thread, as it only blocks in dispatch_semaphore_wait once (waiting for the entire query to complete) and doesn't "do" anything until the query finishes.

Regarding the feedback, I think we'll file a report soon.

Please do and upload full crash logs, posting the number back here once you've got everything uploaded. I'd like to look at a set of full logs to see if I can see anything.

Also, do you have any crash logs without asan? And do they look "the same"? It's not unusual for asan to shift crash behavior (basically, you end up crashing earlier than you normally "would") and it can also "mask" the real crash cause in your app.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

0