How to Symbolicate an Apple Silicon Panic?

Investigating a kernel panic, I discovered that Apple Silicon Panic traces are not working with how I know to symbolicate the panic information. I have not found proper documentation that corrects this situation.

Attached file is an indentity-removed panic, received from causing an intentional panic (dereferencing nullptr), so that I know what functions to expect in the call stack. This is cut-and-pasted from the "Report To Apple" dialog that appears after the reboot:

To start, I download and install the matching KDK (in this case KDK_14.6.1_23G93.kdk), identified from this line:

OS version: 23G93
Kernel version: Darwin Kernel Version 23.6.0: Mon Jul 29 21:14:04 PDT 2024; root:xnu-10063.141.2~1/RELEASE_ARM64_T8122

Then start lldb from Terminal, using this command:

  • bash_prompt % lldb -arch arm64e /Library/Developer/KDKs/KDK_14.6.1_23G93.kdk/System/Library/Kernels/kernel.release.t8122

Next I load the remaining scripts per the instructions from lldb:

  • (lldb) settings set target.load-script-from-symbol-file true

I need to know what address to load my kext symbols to, which I read from this line of the panic log, after the @ symbol:

com.company.product(1.4.21d119)[92BABD94-80A4-3F6D-857A-3240E4DA8009]@0xfffffe001203bfd0->0xfffffe00120533ab

I am using a debug build of my kext, so the DWARF symbols are part of the binary. I use this line to load the symbols into the lldb session:

  • (lldb) addkext -F /Library/Extensions/KextName.kext/Contents/MacOS/KextName 0xfffffe001203bfd0

And now I should be able to use lldb image lookup to identify pointers on the stack that land within my kext. For example, the current PC at the moment of the crash lands within the kext (expected, because it was intentional):

  • (lldb) image lookup -a 0xfffffe001203fe10

Which gives the following incorrect result:

      Address: KextName[0x0000000000003e40] (KextName.__TEXT.__cstring + 14456)
      Summary: "ffer has %d retains\n"

That's not even a program instruction - that's within a cstring. No, that cstring isn't involved in anything pertaining to the intentional panic I am expecting to see.

Can someone please explain what I'm doing wrong and provide instructions that will give symbol information from a panic trace on an Apple Silicon Mac?

Disclaimers:

  • Yes I know IOPCIFamily is deprecated, I am in process of transitioning to DriverKit Dext from IOKit kext. Until then I must maintain the kext.
  • Terminal command "atos" provides similar incorrect results, and seems to not work with debug-built-binaries (only dSYM files)
  • Yes this is an intentional panic so that I can verify the symbolicate process before I move on to investigating an unexpected panic
  • I have set nvram boot-args to include keepsyms=1
  • I have tried (lldb) command script import lldb.macosx but get a result of error: no images in crash log (after the nvram settings)
Answered by daniek3 in 802093022

Kevin Elliot, DTS Engineer, solved this in a private code-level support ticket.

There is yet another offset that was not being accounted for, but can be found using the command line tool otool:

zsh_prompt% otool -arch arm64e -l /Library/Extensions/KEXT_NAME.kext/Contents/MacOS/KEXT_NAME

will give a lot of output information, but a section with this information:

      cmd LC_SEGMENT_64
  cmdsize 232
  segname __TEXT_EXEC
   vmaddr 0x0000000000004000
   vmsize 0x0000000000010000

Shows that an address of 0x4000 needs to be accounted for when performing the atos lookups.

By adding that value to the load address of the kext when calling atos, we get the correct / expected answer.

zsh_prompt% atos -arch arm64e -o KEXT_NAME.kext.dSYM/Contents/Resources/DWARF/KEXT_NAME  -l <load address + (vmaddr of __TEXT_EXEC section)>  0xfffffe0015426e2c

com_company_kext_name::expectedFunction(OSObject*, void*, void*, void*, void*) (in KEXT_NAME) (SourceFile.cpp:1999)

Such a relief! Kudos to Kevin Elliot for finding a solution to this.

First off, please file a bug about the documentation side of this and then post the number back here. We really should do a much better job of explaining exactly what all this involves.

Next, a clarification on this:

That's not even a program instruction - that's within a cstring. No, that cstring isn't involved in anything pertaining to the intentional panic I am expecting to see.

Which gives the following incorrect result:

  Address: KextName[0x0000000000003e40] (KextName.__TEXT.__cstring + 14456)
  Summary: "ffer has %d retains\n"

That's not even a program instruction - that's within a cstring. No, that cstring isn't involved in anything pertaining to the intentional panic I am expecting to see.

What you described here isn't exactly what's going on.

The symbolication process is inherently, for lack of a better word, "stupid". Conceptually, your symbol file is basically just a long list of strings and offset pair, so all symbolication does is find the closest offset "smaller" than the address it was given, then returns "String + remainder". The problem with this line:

  Address: KextName[0x0000000000003e40] (KextName.__TEXT.__cstring + 14456)

...is that you're "remainder" is ENORMOUS. 14456 bytes-> 14Kb. It's not pointing you at a string, it's pointing you at "crazy offset" from some other string. If you work the math out "manually", you can how this all looks very "odd". Here's the basic math:

NOTE: I've thrown out the upper 32 bits, as it makes the numbers easier to work with

Base Address: 0xfffffe001203bfd0 -> 0x1203bfd0

Crashing Address: 0xfffffe001203fe10 -> 0x1203fe10

Crash Offset = Crashing Address - Base Address
0x3E40 = 0x1203fe10 - 0x1203bfd0

Symbolication Offset = "... + 14456" -> 0x3878


Offset of "KextName.__TEXT.__cstring" = Crash Offset - Symbolication Offset

0x5C8 = 0x3E40 - 0x3878

0x5C8 -> 1480 bytes

In other words, the actual location of "KextName.__TEXT.__cstring" is 1480 bytes from the "start" of your library, which actually seems somewhat reasonable.

In other words, the question here isn't "why am I pointing at a c string", it's "why aren't there any other symbols in the ~13,000 bytes after that cstring". Here's what I'd actually do next:

  • Symbolicate the other frames you've got. This is a quick way to differentiate between "something is wrong with THIS symbol" vs "something systemic is wrong".

  • "dwarfdump" will print the contents of a dsym file, which will either show that the symbols

  • Assuming there isn't any issue with the symbol file itself, then this could be because ASLR side your KEXT and hasn't been accounted for correctly. I think that's what this slide actually is:

KernelCache slide: 0x000000000b1c0000

...but I'm not sure if that was already accounted for in your addresses or not. However, if your dsym is otherwise "valid", then you should also be able to see a difference between the offset the kernel returned for the symbol it did find (KextName.__TEXT.__cstring-> 0x5C8) and the offset the dsym lists for that symbol. You can then adjust your address offsets to remove that difference.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Documentation bug reported, as FB14929037.

Yes - the offset from the string is quite large, but I just want useful output such as "ThisFunctionWasInvolved" from "ThisFunctionCausedAPanic.cpp, line 455".

Symbolicate the other frames you've got. This is a quick way to differentiate between "something is wrong with THIS symbol" vs "something systemic is wrong".

Yes, this is a “something systemic is wrong” situation. I was trying to reduce the amount of information for readers to wade through, but I am not getting useful results for any efforts at symbolicating the addresses on the back trace or the PC (program counter).

” dwarfdump" will print the contents of a dsym file, which will either show that the symbols

dwarfdump works on dSYM files, but apparently not on debug-builds-with-symbols-in-the-binary. As such, I rebuilt with separating the symbols into a dSYM file. dwarfdump provides what appears to be correct information for the symbols, but for a sanity check here is a blurb from the start of the dump:

KEXT_NAME.kext.dSYM/Contents/Resources/DWARF/KEXT_NAME(arm64e):	file format Mach-O arm64

.debug_info contents:
0x00000000: Compile Unit: length = 0x000001f9, format = DWARF32, version = 0x0004, abbr_offset = 0x0000, addr_size = 0x08 (next unit at 0x000001fd)

0x0000000b: DW_TAG_compile_unit
              DW_AT_producer	("Apple clang version 13.1.6 (clang-1316.0.21.2.5)")
              DW_AT_language	(DW_LANG_C99)
              DW_AT_name	("/Users/username/Library/Developer/Xcode/DerivedData/KEXT_NAME-stuff/ArchiveIntermediates/KEXT_NAME/IntermediateBuildFilesPath/KEXT_NAME.build/Release/KEXT_NAME.build/DerivedSources/KEXT_NAME_info.c")
              DW_AT_LLVM_sysroot	("/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk")
              DW_AT_APPLE_sdk	("MacOSX12.3.sdk")
              DW_AT_stmt_list	(0x00000000)
              DW_AT_comp_dir	("/Users/username/sandbox/codedirectory")
              DW_AT_APPLE_optimized	(true)

Since the binary is different, I triggered a new panic to work with:

The command line utility atos should be helpful as well, but here’s an example of what I get as output from that, when using full dSYM file information:

zsh_prompt% atos -arch arm64e -o KEXT_NAME.kext.dSYM/Contents/Resources/DWARF/KEXT_NAME  -l 0xfffffe0015423fd0 0xfffffe0015426e2c
0x00002e5c (in KEXT_NAME)

If I manually look up the 0x2e5c offset in the output from dwarfdump, I can see the following:

0x00002e5c:       DW_TAG_formal_parameter
                    DW_AT_type	(0x0000000000009408 "const IOService *")
                    DW_AT_artificial	(true) 

But that's definitely not conveying the one-step-to-symbolication that atos is supposed to be able to perform. It does not appear that I am using it incorrectly, but I am definitely not getting helpful results from it.

Following the idea that perhaps the kernel slide is not being properly dealt with, I followed this link ( https://lists.apple.com/archives/darwin-kernel/2014/Jan/msg00011.html ) for advice on how to deal with kernel slide (below is all commands listed, and lack of useful information provided):

So to answer all of the questions:

  1. Systemic issue, I am having trouble interpreting a kernel panic log and getting valid symbols as a result of address lookup for Apple Silicon
  2. dSYM file appears to be built properly
  3. atos is not providing helpful information, despite having proper dSYM file
  4. lldb commands to load images and deal with kernel slides does not yield helpful information
Accepted Answer

Kevin Elliot, DTS Engineer, solved this in a private code-level support ticket.

There is yet another offset that was not being accounted for, but can be found using the command line tool otool:

zsh_prompt% otool -arch arm64e -l /Library/Extensions/KEXT_NAME.kext/Contents/MacOS/KEXT_NAME

will give a lot of output information, but a section with this information:

      cmd LC_SEGMENT_64
  cmdsize 232
  segname __TEXT_EXEC
   vmaddr 0x0000000000004000
   vmsize 0x0000000000010000

Shows that an address of 0x4000 needs to be accounted for when performing the atos lookups.

By adding that value to the load address of the kext when calling atos, we get the correct / expected answer.

zsh_prompt% atos -arch arm64e -o KEXT_NAME.kext.dSYM/Contents/Resources/DWARF/KEXT_NAME  -l <load address + (vmaddr of __TEXT_EXEC section)>  0xfffffe0015426e2c

com_company_kext_name::expectedFunction(OSObject*, void*, void*, void*, void*) (in KEXT_NAME) (SourceFile.cpp:1999)

Such a relief! Kudos to Kevin Elliot for finding a solution to this.

Kudos to Kevin Elliot for finding a solution to this.

You're very welcome. atos needs to be updated to address this, probably by adding a new option when importing the base address. However, until then the math above should work fine.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

How to Symbolicate an Apple Silicon Panic?
 
 
Q