You’re now watching this thread. If you’ve opted in to email or web notifications, you’ll be notified when there’s activity. Click again to stop watching or visit your profile to manage watched threads and notifications.
You’ve stopped watching this thread and will no longer receive emails or web notifications when there’s activity. Click again to start watching.
Investigating a kernel panic, I discovered that Apple Silicon Panic traces are not working with how I know to symbolicate the panic information. I have not found proper documentation that corrects this situation.
Attached file is an indentity-removed panic, received from causing an intentional panic (dereferencing nullptr), so that I know what functions to expect in the call stack. This is cut-and-pasted from the "Report To Apple" dialog that appears after the reboot:
And now I should be able to use lldb image lookup to identify pointers on the stack that land within my kext. For example, the current PC at the moment of the crash lands within the kext (expected, because it was intentional):
(lldb) image lookup -a 0xfffffe001203fe10
Which gives the following incorrect result:
Address: KextName[0x0000000000003e40] (KextName.__TEXT.__cstring + 14456)
Summary: "ffer has %d retains\n"
That's not even a program instruction - that's within a cstring. No, that cstring isn't involved in anything pertaining to the intentional panic I am expecting to see.
Can someone please explain what I'm doing wrong and provide instructions that will give symbol information from a panic trace on an Apple Silicon Mac?
Disclaimers:
Yes I know IOPCIFamily is deprecated, I am in process of transitioning to DriverKit Dext from IOKit kext. Until then I must maintain the kext.
Terminal command "atos" provides similar incorrect results, and seems to not work with debug-built-binaries (only dSYM files)
Yes this is an intentional panic so that I can verify the symbolicate process before I move on to investigating an unexpected panic
I have set nvram boot-args to include keepsyms=1
I have tried (lldb) command script import lldb.macosx but get a result of error: no images in crash log (after the nvram settings)
First off, please file a bug about the documentation side of this and then post the number back here. We really should do a much better job of explaining exactly what all this involves.
Next, a clarification on this:
That's not even a program instruction - that's within a cstring. No, that cstring isn't involved in anything pertaining to the intentional panic I am expecting to see.
Which gives the following incorrect result:
Address: KextName[0x0000000000003e40] (KextName.__TEXT.__cstring + 14456)
Summary: "ffer has %d retains\n"
That's not even a program instruction - that's within a cstring. No, that cstring isn't involved in anything pertaining to the intentional panic I am expecting to see.
What you described here isn't exactly what's going on.
The symbolication process is inherently, for lack of a better word, "stupid". Conceptually, your symbol file is basically just a long list of strings and offset pair, so all symbolication does is find the closest offset "smaller" than the address it was given, then returns "String + remainder". The problem with this line:
...is that you're "remainder" is ENORMOUS. 14456 bytes-> 14Kb. It's not pointing you at a string, it's pointing you at "crazy offset" from some other string. If you work the math out "manually", you can how this all looks very "odd". Here's the basic math:
NOTE: I've thrown out the upper 32 bits, as it makes the numbers easier to work with
Base Address: 0xfffffe001203bfd0 -> 0x1203bfd0
Crashing Address: 0xfffffe001203fe10 -> 0x1203fe10
Crash Offset = Crashing Address - Base Address
0x3E40 = 0x1203fe10 - 0x1203bfd0
Symbolication Offset = "... + 14456" -> 0x3878
Offset of "KextName.__TEXT.__cstring" = Crash Offset - Symbolication Offset
0x5C8 = 0x3E40 - 0x3878
0x5C8 -> 1480 bytes
In other words, the actual location of "KextName.__TEXT.__cstring" is 1480 bytes from the "start" of your library, which actually seems somewhat reasonable.
In other words, the question here isn't "why am I pointing at a c string", it's "why aren't there any other symbols in the ~13,000 bytes after that cstring". Here's what I'd actually do next:
Symbolicate the other frames you've got. This is a quick way to differentiate between "something is wrong with THIS symbol" vs "something systemic is wrong".
"dwarfdump" will print the contents of a dsym file, which will either show that the symbols
Assuming there isn't any issue with the symbol file itself, then this could be because ASLR side your KEXT and hasn't been accounted for correctly. I think that's what this slide actually is:
KernelCache slide: 0x000000000b1c0000
...but I'm not sure if that was already accounted for in your addresses or not. However, if your dsym is otherwise "valid", then you should also be able to see a difference between the offset the kernel returned for the symbol it did find (KextName.__TEXT.__cstring-> 0x5C8) and the offset the dsym lists for that symbol. You can then adjust your address offsets to remove that difference.
Yes - the offset from the string is quite large, but I just want useful output such as "ThisFunctionWasInvolved" from "ThisFunctionCausedAPanic.cpp, line 455".
Symbolicate the other frames you've got. This is a quick way to differentiate between "something is wrong with THIS symbol" vs "something systemic is wrong".
Yes, this is a “something systemic is wrong” situation. I was trying to reduce the amount of information for readers to wade through, but I am not getting useful results for any efforts at symbolicating the addresses on the back trace or the PC (program counter).
” dwarfdump" will print the contents of a dsym file, which will either show that the symbols
dwarfdump works on dSYM files, but apparently not on debug-builds-with-symbols-in-the-binary. As such, I rebuilt with separating the symbols into a dSYM file. dwarfdump provides what appears to be correct information for the symbols, but for a sanity check here is a blurb from the start of the dump:
KEXT_NAME.kext.dSYM/Contents/Resources/DWARF/KEXT_NAME(arm64e): file format Mach-O arm64
.debug_info contents:
0x00000000: Compile Unit: length = 0x000001f9, format = DWARF32, version = 0x0004, abbr_offset = 0x0000, addr_size = 0x08 (next unit at 0x000001fd)
0x0000000b: DW_TAG_compile_unit
DW_AT_producer ("Apple clang version 13.1.6 (clang-1316.0.21.2.5)")
DW_AT_language (DW_LANG_C99)
DW_AT_name ("/Users/username/Library/Developer/Xcode/DerivedData/KEXT_NAME-stuff/ArchiveIntermediates/KEXT_NAME/IntermediateBuildFilesPath/KEXT_NAME.build/Release/KEXT_NAME.build/DerivedSources/KEXT_NAME_info.c")
DW_AT_LLVM_sysroot ("/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk")
DW_AT_APPLE_sdk ("MacOSX12.3.sdk")
DW_AT_stmt_list (0x00000000)
DW_AT_comp_dir ("/Users/username/sandbox/codedirectory")
DW_AT_APPLE_optimized (true)
Since the binary is different, I triggered a new panic to work with:
The command line utility atos should be helpful as well, but here’s an example of what I get as output from that, when using full dSYM file information:
But that's definitely not conveying the one-step-to-symbolication that atos is supposed to be able to perform. It does not appear that I am using it incorrectly, but I am definitely not getting helpful results from it.
Following the idea that perhaps the kernel slide is not being properly dealt with, I followed this link ( https://lists.apple.com/archives/darwin-kernel/2014/Jan/msg00011.html ) for advice on how to deal with kernel slide (below is all commands listed, and lack of useful information provided):
zsh_prompt % lldb -arch arm64e kernel.release.t8122
(lldb) target create --arch=arm64e "kernel.release.t8122"
warning: 'kernel.release' contains a debug script. To run this script in this debug session:
command script import "/Library/Developer/KDKs/KDK_14.6.1_23G93.kdk/System/Library/Kernels/kernel.release.t8122.dSYM/Contents/Resources/Python/kernel_release.py"
To run all discovered debug scripts in this session:
settings set target.load-script-from-symbol-file true
Current executable set to '/Library/Developer/KDKs/KDK_14.6.1_23G93.kdk/System/Library/Kernels/kernel.release.t8122' (arm64e).
(lldb) settings set target.load-script-from-symbol-file true
Loading kernel debugging from /Library/Developer/KDKs/KDK_14.6.1_23G93.kdk/System/Library/Kernels/kernel.release.t8122.dSYM/Contents/Resources/Python/kernel_release.py
LLDB version lldb-1500.0.404.7
Apple Swift version 5.10 (swiftlang-5.10.0.13 clang-1500.3.9.4)
settings set target.process.python-os-plugin-path "/Library/Developer/KDKs/KDK_14.6.1_23G93.kdk/System/Library/Kernels/kernel.release.t8122.dSYM/Contents/Resources/Python/lldbmacros/core/operating_system.py"
settings set target.trap-handler-names hndl_allintrs hndl_alltraps trap_from_kernel hndl_double_fault hndl_machine_check _fleh_prefabt _ExceptionVectorsBase _ExceptionVectorsTable _fleh_undef _fleh_dataabt _fleh_irq _fleh_decirq _fleh_fiq_generic _fleh_dec
command script import "/Library/Developer/KDKs/KDK_14.6.1_23G93.kdk/System/Library/Kernels/kernel.release.t8122.dSYM/Contents/Resources/Python/lldbmacros/xnu.py"
xnu debug macros loaded successfully. Run showlldbtypesummaries to enable type summaries.
settings set target.process.optimization-warnings false
settings set target.process.experimental.os-plugin-reports-all-threads false
settings set target.process.run-all-threads true
(lldb) target modules add ../Extensions/IOPCIFamily.kext/Contents/MacOS/IOPCIFamily
(lldb) target modules add /Library/Extensions/KEXT_NAME.kext/Contents/MacOS/KEXT_NAME
(lldb) target modules load --file kernel.release.t8122 --slide 0x000000000e5a4000
Loading kernel debugging from /Library/Developer/KDKs/KDK_14.6.1_23G93.kdk/System/Library/Kernels/kernel.release.t8122.dSYM/Contents/Resources/Python/kernel_release.py
LLDB version lldb-1500.0.404.7
Apple Swift version 5.10 (swiftlang-5.10.0.13 clang-1500.3.9.4)
settings set target.process.python-os-plugin-path "/Library/Developer/KDKs/KDK_14.6.1_23G93.kdk/System/Library/Kernels/kernel.release.t8122.dSYM/Contents/Resources/Python/lldbmacros/core/operating_system.py"
settings set target.trap-handler-names hndl_allintrs hndl_alltraps trap_from_kernel hndl_double_fault hndl_machine_check _fleh_prefabt _ExceptionVectorsBase _ExceptionVectorsTable _fleh_undef _fleh_dataabt _fleh_irq _fleh_decirq _fleh_fiq_generic _fleh_dec
command script import "/Library/Developer/KDKs/KDK_14.6.1_23G93.kdk/System/Library/Kernels/kernel.release.t8122.dSYM/Contents/Resources/Python/lldbmacros/xnu.py"
error: cannot add command: user command exists and force replace not set
error: cannot add command: user command exists and force replace not set
error: cannot add command: user command exists and force replace not set
error: cannot add command: user command exists and force replace not set
error: cannot add command: user command exists and force replace not set
error: cannot add command: user command exists and force replace not set
xnu debug macros loaded successfully. Run showlldbtypesummaries to enable type summaries.
error: cannot add command: user command exists and force replace not set
error: cannot add command: user command exists and force replace not set
error: cannot add command: user command exists and force replace not set
error: cannot add command: user command exists and force replace not set
error: cannot add command: user command exists and force replace not set
error: cannot add command: user command exists and force replace not set
error: cannot add command: user command exists and force replace not set
settings set target.process.optimization-warnings false
settings set target.process.experimental.os-plugin-reports-all-threads false
settings set target.process.run-all-threads true
(lldb) target modules load --file IOPCIFamily __TEXT 0xfffffe0019241e90
section '__TEXT' loaded at 0xfffffe0019241e90
(lldb) target modules load --file KEXT_NAME __TEXT 0xfffffe0015423fd0
section '__TEXT' loaded at 0xfffffe0015423fd0
(lldb) image list
[ 0] 5ABEB946-A4FE-30C7-AD57-9F34F3E88710 0xfffffe00155a8000 /Library/Developer/KDKs/KDK_14.6.1_23G93.kdk/System/Library/Kernels/kernel.release.t8122
/Library/Developer/KDKs/KDK_14.6.1_23G93.kdk/System/Library/Kernels/kernel.release.t8122.dSYM/Contents/Resources/DWARF/kernel.release.t8122
[ 1] 6D6666E6-340F-3A5E-9464-DE05164C0658 0xfffffe0019241e90 ../Extensions/IOPCIFamily.kext/Contents/MacOS/IOPCIFamily
../Extensions/IOPCIFamily.kext.dSYM/Contents/Resources/DWARF/IOPCIFamily
[ 2] BCD984AA-BC5B-30C7-94A7-4065678A0438 0xfffffe0015423fd0 /Library/Extensions/KEXT_NAME.kext/Contents/MacOS/KEXT_NAME
/Library/Extensions/KEXT_NAME.kext.dSYM/Contents/Resources/DWARF/KEXT_NAME
(lldb) image lookup -a 0xfffffe0015426bcc
Address: KEXT_NAME[0x0000000000002bfc] (KEXT_NAME.__TEXT + 11260)
Summary:
(lldb) image lookup -a 0xfffffe0015426c50
Address: KEXT_NAME[0x0000000000002c80] (KEXT_NAME.__TEXT + 11392)
Summary:
(lldb) image lookup -a 0xfffffe0015426e2c
Address: KEXT_NAME[0x0000000000002e5c] (KEXT_NAME.__TEXT + 11868)
Summary:
So to answer all of the questions:
Systemic issue, I am having trouble interpreting a kernel panic log and getting valid symbols as a result of address lookup for Apple Silicon
dSYM file appears to be built properly
atos is not providing helpful information, despite having proper dSYM file
lldb commands to load images and deal with kernel slides does not yield helpful information
Kudos to Kevin Elliot for finding a solution to this.
You're very welcome. atos needs to be updated to address this, probably by adding a new option when importing the base address. However, until then the math above should work fine.
