Instructions for debugging recent macos kernel versions?

Is there any recent and a bit authoritative documentation which explains how to debug recent versions of macos kernel?

No, nothing for arm* that's directly published by Apple. However, I'm told by an engineer on the kernel team that the arm install instructions here "work":

https://kernelshaman.blogspot.com/2021/02/building-xnu-for-macos-112-intel-apple.html

*The process for Intel hasn't really changed.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

However, I'm told by an engineer on the kernel team that the arm install instructions here "work":

https://kernelshaman.blogspot.com/2021/02/building-xnu-for-macos-112-intel-apple.html

Thank you Kevin, I'll give that a try in the coming days. As for x64 vs arm, I am interested in arm since that's what I have access to.

Short update - the linked article did help me successfully build a kext collection for the kernel on my local M2 system. I had to do minor adjustments to the instructions in that article to get it working.

I have been trying to boot my M2 using this kext collection (after "kmutil configure-boot ..." successfully completed), but so far that step hasn't worked. Every time the system tries to boot with this custom built kext collection (in Permissive mode), the Apple icon shows up for a few seconds and then system shutdown and then again the Apple icon shows up and this keeps continuing. I switched back to "Restrictive mode" to allow me to boot the regular kernel, but I'll experiment some more and see how I can debug/investigate what's preventing a successful boot - it's going to be challenging because I can't seem to locate any logs for this part of the boot process.

I have been trying to boot my M2 using this kext collection (after "kmutil configure-boot ..." successfully completed), but so far that step hasn't worked. Every time the system tries to boot with this custom built kext collection (in Permissive mode), the Apple icon shows up for a few seconds and then system shutdown and then again the Apple icon shows up and this keeps continuing.

One quick point here- make sure you're using EXACTLY the correct KDK version and development kernel version. Running "unname -a" on your working system will get you the right T* kernel variant.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Hello Kevin,

"uname -v" on my local system shows:

Darwin Kernel Version 24.6.0: Mon Jul 14 11:30:30 PDT 2025; root:xnu-11417.140.69~1/RELEASE_ARM64_T6020

so I've been using T6020 as the "platform" when building the kernel.

The corresponding xnu source is this tag https://github.com/apple-oss-distributions/xnu/tree/xnu-11417.140.69 and the KDK is KDK_15.6_24G84.kdk. sw_vers shows me:

sw_vers
ProductName:		macOS
ProductVersion:		15.6
BuildVersion:		24G84

So I think that's the right KDK version for this system.

I'm guessing the XCode version doesn't matter for these experiments (as long as the build succeeds), but for the record, I'm on XCode 16.4:

xcodebuild -version
Xcode 16.4
Build version 16F6

Apart from the other build instructions, the primary one I use to generate a "DEVELOPMENT" variant of the kernel is:

export KDK=/Library/Developer/KDKs/KDK_15.6_24G84.kdk
make SDKROOT=macosx KDKROOT=$KDK BUILD_WERROR=0 BUILD_JSON_COMPILATION_DATABASE=1 TARGET_CONFIGS="DEVELOPMENT ARM64 T6020"

and then to generate a (bootable) kext collection for this kernel, I use (this command is similar to what's noted in the linked article):

kmutil create \
        --arch arm64e \
        --no-authorization \
        --variant-suffix development \
        --new boot \
        --boot-path /Users/me/xnu-11417.140.69/dist/macos-15.6-development.kc \
        --kernel /Users/me/xnu-11417.140.69/dist/kernel.development.T6020 \
        --repository $KDK/System/Library/Extensions \
        --repository /System/Library/Extensions \
        --repository /System/Library/DriverExtensions \
        --explicit-only $(kmutil inspect -V release --no-header | grep -v "SEPHiber" | awk '{print " -b "$1; }')

This generates the kext collection at /Users/me/xnu-11417.140.69/dist/macos-15.6-development.kc.development and the following command against that kext collection file seems to suggest all looks OK:

file  /Users/me/xnu-11417.140.69/dist/macos-15.6-development.kc.development

/Users/me/xnu-11417.140.69/dist/macos-15.6-development.kc.development: Mach-O 64-bit arm64e

All this goes fine, so does running the following command in recovery mode:

kmutil configure-boot --volume "/Volumes/Macintosh HD" --custom-boot-object "/Volumes/Macintosh HD/Users/me/xnu-11417.140.69/dist/macos-15.6-development.kc.development"

(along with "nvram boot-args='-v wlan.skywalk.enable=0 dk=0'")

Yet, when I restart with that configured custom boot object, the Apple icon keep repeating and the system keeps restarting. I haven't done any changes to the kernel code itself, so I can rule out any custom code contributing to this.

My next plan is to build the RELEASE variant of this kernel and see if that boots fine instead of the DEVELOPMENT variant. If not, I'll read up a bit to understand how to track down the boot issue. Thank you very much for the helpful hints so far.

so I've been using T6020 as the "platform" when building the kernel.

OK, so this is the part I haven't explained as well as I should. You don't actually need to build the kernel, as you already have a development build of the kernel in the KDK. That's what "kernel.development.t6020" and "kernel.kasan.t6020" installed by the KDK in /System/Library/Kernels "are".

You should be able to "jump" straight to here:

and then to generate a (bootable) kext collection for this kernel, I use (this command is similar to what's noted in the linked article):

...and continue from that point.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

OK, so this is the part I haven't explained as well as I should. You don't actually need to build the kernel, as you already have a development build of the kernel in the KDK. That's what "kernel.development.t6020" and "kernel.kasan.t6020" installed by the KDK in /System/Library/Kernels "are"

That was it - I built a kext collection using this pre-shipped kernels in the KDK and I was then able to boot this "custom" kext collection without any issues. I used the kernel.development variant for my test. Upon boot I even verified that it was indeed using the development variant of the kernel:

$> sysctl kern.osbuildconfig
kern.osbuildconfig: development

Thank you very much Kevin for this very valuable help.

I have several other things to read and try out (including hopefully building a trivially modified kernel version from xnu sources and booting it), but this initial step had blocked me for several months to even think of these additional experiments.

For now, I will just write up these steps and share here. Thank you again.

That was it - I built a kext collection using these pre-shipped kernels in the KDK and I was then able to boot this "custom" kext collection without any issues. I used the kernel.development variant for my test. Upon boot, I even verified that it was indeed using the development variant of the kernel:

Fabulous!

(including hopefully building a trivially modified kernel version from xnu sources and booting it)

It's been a very long time since I've done this, but the main thing I remember is that you need to be very careful to ensure that you've exactly matched up versions so that "everything" is the same exact version (kernel source, KDK, developer tools, etc.). The problem isn't so much that it MUST be the same, it's that there are so many details and "every" failure basically looks the same ("it doesn't boot!"). That makes it really easy to waste a ton of time trying to hunt down small "problems" that are basically just noise.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

For now, I will just write up these steps and share here.

For future reference, I've now written it up here https://jaitechwriteups.blogspot.com/2025/10/boot-custom-macos-kernel-on-macos-apple.html

It's been a very long time since I've done this, but the main thing I remember is that you need to be very careful to ensure that you've exactly matched up versions so that "everything" is the same exact version (kernel source, KDK, developer tools, etc.).

Thank you Kevin for that hint. I suspected it wouldn't be straightforward. It's still on my TODO list to experiment with that and hopefully get it to work in the coming weeks. If it does, then it will help in a big way in debugging (and reporting) some of the issues we have been running into recently.

Hi hi,

I'm having issues with this as well. I found this forum post through @jaikiran's blog post, which has been of massive help.

I just tried this method on a mac mini M2 Pro (macOS 15.7.3 build 24G419 and T6020) and it worked absolutely fine on the first try. When I tried it on a new mac mini M6 (27.0.1 build 26A434, T8152), macOS can't boot and at some point it just panics into recovery mode.

Did something change within the last year that I have to look out for?

Many thanks in advance and best,

Andres

Did something change within the last year that I have to look out for?

I'm not sure. Have you filed a bug with the panic log and, if so, what's the bug number? Also, did you try both kernels (DEVELOPMENT and DEBUG)?

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Hello Andres,

When I tried it on a new mac mini M6 (27.0.1 build 26A434, T8152), macOS can't boot and at some point it just panics into recovery mode.

In addition to what Kevin noted in the reply, what is the output of uname -v on that host and what is the KDK build that you downloaded for it? Also, what is the name of the kernel binary file (the one from the downloaded KDK, which contains the model name) you used to generate the kext collection?

Hi to both, thanks for your responses.

I've been trying multiple permutations to see what would work. I still haven't managed much that I could consider helpful.

With the commands to create the .kc files, I made two from the release kernel with suffixes for development and debug, and the same for the development kernel. In the end I had files for release.kc.development, release.kc.debug, dev.kc.development, dev.kc.debug.

Both release versions booted no prob, the dev ones showed the same issue, where the mac sound gets played, then it connects to the network (either DCHP or a manually set one), but it doesn't take any input from external devices and displays nothing.

From there it seems to reboot itself a bunch of time, as the network connection stops working multiple times until it eventually "breaks" and forces me into a window with "The version of macOS on the selected disk needs to be reinstalled" and the option to go to recovery.

In addition to what Kevin noted in the reply, what is the output of uname -v on that host and what is the KDK build that you downloaded for it? Also, what is the name of the kernel binary file (the one from the downloaded KDK, which contains the model name) you used to generate the kext collection?

uname -v
Darwin Kernel Version 27.0.0: Tue Aug 11 21:06:15 PDT 2026; root:xnu-13432.1.9~1/RELEASE_ARM64_T8152

sw_vers
ProductName:		macOS
ProductVersion:		27.0.1
BuildVersion:		26A434

I'm using KDK_27.0.1_26A434.kdk, which matches the product and build versions.

Ever since you guys' responses I've been trying to generate reports to see if I could report a bug with some substance.

Debugging hasn't really worked as well as I hoped, but since I don't have a working system I can try, I don't know if maybe I'm missing something. Basically I didn't manage to do a network debug, neither connecting two computers directly via ethernet and setting static addresses nor connecting them to a switch with DHCP worked. I thought it might be the fact that I couldn't use USB ethernet adapters, but then I borrowed another mac mini again to try it out with both ethernet ports and nothing worked anyways.

I looked into some older articles online about debugging where I could use the power button to trigger an NMI to dump the file directly into the device, without needing a second one. That also didn't work. I think I was looking for the files in the "wrong" directory (/var/tmp/kernel_panics). I was later able to find some stuff under /Library/Logs/DiagnosticReports.

At some point after the message about macOS being broken, I could open recovery and the terminal to see an .ips file. After rebooting with the high security mode in order to boot normally, the file wasn't there anymore. This is one of the refs I used the most and that seemed the most complete but nothing from there really worked: https://karol-mazurek.medium.com/kernel-debugging-setup-on-macos-07dd8c86cdb6

I replicated the steps I took to recreate and copy the .ips file to another place where it would persist, but I couldn't get it to appear anymore. Otherwise I was able to see some .shutdownStall files. I made the latest one readable, but based on the "Posting a crash report" post don't know if it's a relevant file since it's neither a crash nor an ips. It's got stuff about pci errors and drivers.

Any advice on how to proceed would be appreciated. I'm kind of at my wit's end.

Many thanks!

Any advice on how to proceed would be appreciated. I'm kind of at my wit's end.

SO, I'm afraid the answer here is basically, "yes, it's broken.” You can (and should) file a bug on this, but I wouldn't be surprised if it takes a little while to get this working.

What's actually going on here is that, as you might imagine, every system build actually involves creating several different versions of the system, part of which is the variant specifically focused on "development.” Unsurprisingly, the development kernel in the KDK is actually extracted from the larger development configuration.

In any case, while this should work, if the right/wrong things change across variants, you can end up in a situation where the development kernel panics on the release system when it tries to access data/resources/infrastructure that simply don't exist on the release system. Those failures are bugs (and if you haven't already, please do file a bug on this), but they won't prevent us from releasing the system (because the overall system works fine) or KDK. That last point is also why fixing this may take a little while as these kind of changes are inherently high risk (there's no such thing as a "safe" kernel patch), low benefit (this particular class of panic's is not one a typical end user will ever experience), which means they'll need to go through additional time and testing before they're released.

Unfortunately, all of that means that all I can really suggest is to test this again on future releases and not waste too much time beyond that.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Hi, I can confirm that we received a bug on this issue, and that the right folks are looking into a fix. The TL;DR is that M5 and M6 Macs are going to panic on boot if you link a kernel cache with the development kernel from the KDK or you build your own custom development kernel from the sources because of an unforseen incompatibility with the release firmware present on the M5 and M6 Macs. M4 and earlier are reported to still work according to the reporter.

For now, I will just write up these steps and share here.

For future reference, I've now written it up here https://jaitechwriteups.blogspot.com/2025/10/boot-custom-macos-kernel-on-macos-apple.html

Hi, nice to meet you jaikiran! I'm the maintainer within the kernel team of the opensource XNU release, and I have tried to put as much of the instructions needed into the README.md in the more recent releases as possible. I realized about six months ago that the README.md itself would be a reasonable place for instructions on how to build it, but for some reason it was only being used as an internal file within XNU at that point (and even then I'm not sure people read it, haha). Probably nobody noticed, so if you have any feedback for improvement feel free to send it my way and I'll gladly update it.

Hi everyone. Thank you so much for your support on this. I had to take a bit of a break from the subject to see if I could work on it from a clean slate.

I was able to get an IPS with the Mac going into recovery mode with PanicMedic as the reason. @elowe, I'll post this as a bug, but I assume that it might be redundant if someone already reported it.

I would also suggest adding some additional debug flag combinations to the README, based on this https://developer.apple.com/library/archive/technotes/tn2004/tn2118.html#SECDEBUGFLAGS which I took from the blog post in my last reply. There's a table with useful flag combinations, which I think could be helpful for further troubleshooting.

Thanks again! :)