Investigating a kernel panic, I discovered that Apple Silicon Panic traces are not working with how I know to symbolicate the panic information. I have not found proper documentation that corrects this situation.
Attached file is an indentity-removed panic, received from causing an intentional panic (dereferencing nullptr), so that I know what functions to expect in the call stack. This is cut-and-pasted from the "Report To Apple" dialog that appears after the reboot:
To start, I download and install the matching KDK (in this case KDK_14.6.1_23G93.kdk), identified from this line:
OS version: 23G93
Kernel version: Darwin Kernel Version 23.6.0: Mon Jul 29 21:14:04 PDT 2024; root:xnu-10063.141.2~1/RELEASE_ARM64_T8122
Then start lldb from Terminal, using this command:
bash_prompt % lldb -arch arm64e /Library/Developer/KDKs/KDK_14.6.1_23G93.kdk/System/Library/Kernels/kernel.release.t8122
Next I load the remaining scripts per the instructions from lldb:
(lldb) settings set target.load-script-from-symbol-file true
I need to know what address to load my kext symbols to, which I read from this line of the panic log, after the @ symbol:
com.company.product(1.4.21d119)[92BABD94-80A4-3F6D-857A-3240E4DA8009]@0xfffffe001203bfd0->0xfffffe00120533ab
I am using a debug build of my kext, so the DWARF symbols are part of the binary. I use this line to load the symbols into the lldb session:
(lldb) addkext -F /Library/Extensions/KextName.kext/Contents/MacOS/KextName 0xfffffe001203bfd0
And now I should be able to use lldb image lookup to identify pointers on the stack that land within my kext. For example, the current PC at the moment of the crash lands within the kext (expected, because it was intentional):
(lldb) image lookup -a 0xfffffe001203fe10
Which gives the following incorrect result:
Address: KextName[0x0000000000003e40] (KextName.__TEXT.__cstring + 14456)
Summary: "ffer has %d retains\n"
That's not even a program instruction - that's within a cstring. No, that cstring isn't involved in anything pertaining to the intentional panic I am expecting to see.
Can someone please explain what I'm doing wrong and provide instructions that will give symbol information from a panic trace on an Apple Silicon Mac?
Disclaimers:
- Yes I know IOPCIFamily is deprecated, I am in process of transitioning to DriverKit Dext from IOKit kext. Until then I must maintain the kext.
- Terminal command "atos" provides similar incorrect results, and seems to not work with debug-built-binaries (only dSYM files)
- Yes this is an intentional panic so that I can verify the symbolicate process before I move on to investigating an unexpected panic
- I have set nvram boot-args to include keepsyms=1
- I have tried (lldb) command script import lldb.macosx but get a result of error: no images in crash log (after the nvram settings)
Kevin Elliot, DTS Engineer, solved this in a private code-level support ticket.
There is yet another offset that was not being accounted for, but can be found using the command line tool otool:
zsh_prompt% otool -arch arm64e -l /Library/Extensions/KEXT_NAME.kext/Contents/MacOS/KEXT_NAME
will give a lot of output information, but a section with this information:
cmd LC_SEGMENT_64
cmdsize 232
segname __TEXT_EXEC
vmaddr 0x0000000000004000
vmsize 0x0000000000010000
Shows that an address of 0x4000 needs to be accounted for when performing the atos lookups.
By adding that value to the load address of the kext when calling atos, we get the correct / expected answer.
zsh_prompt% atos -arch arm64e -o KEXT_NAME.kext.dSYM/Contents/Resources/DWARF/KEXT_NAME -l <load address + (vmaddr of __TEXT_EXEC section)> 0xfffffe0015426e2c
com_company_kext_name::expectedFunction(OSObject*, void*, void*, void*, void*) (in KEXT_NAME) (SourceFile.cpp:1999)
Such a relief! Kudos to Kevin Elliot for finding a solution to this.