On modern systems all KEXTs must be code signed with a Developer ID. Additionally, the Developer ID must be specifically enabled for KEXT development. You can learn more about that process on the Developer ID page.
If your KEXT is having code signing problems, check that it’s signed with a KEXT-enabled Developer ID. Do this by looking at the certificate used to sign the KEXT. First, extract the certificates from the signed KEXT:
% codesign -d --extract-certificates MyKEXT.kext
Executable=/Users/quinn/Desktop/MyKEXT/build/Debug/MyKEXT.kext/Contents/MacOS/MyKEXT
This creates a bunch of certificates of the form codesignNNN, where NNN is a number in the range from 0 (the leaf) to N (the root). For example:
% ls -lh codesign*
-rw-r--r--+ 1 quinn staff 1.4K 20 Jul 10:23 codesign0
-rw-r--r--+ 1 quinn staff 1.0K 20 Jul 10:23 codesign1
-rw-r--r--+ 1 quinn staff 1.2K 20 Jul 10:23 codesign2
Next, rename each of those certificates to include the .cer extension:
% for i in codesign*; do mv $i $i.cer; done
Finally, look at the leaf certificate (codesign0.cer) to see if it has an extension with the OID 1.2.840.113635.100.6.1.18. The easiest way to view the certificate is to use Quick Look in Finder.
Note If you’re curious where these Apple-specific OIDs comes from, check out the documents on the Apple PKI page. In this specific case, look at section 4.11.3 Application and Kernel Extension Code Signing
Certificates of the Developer ID CPS.
If the certificate does have this extension, there’s some other problems with your KEXT’s code signing. In that case, feel free to create a new thread here on DevForums with your details.
If the certificate does not have this extension, there are two possible causes:
Xcode might be using an out-of-date signing certificate. Re-create your Developer ID signing certificate using the developer site and see if the extension shows up there. If so, you’ll have to investigate why Xcode is not using the most up-to-date signing certificate.
If a freshly-created Developer ID signing certificate does not have this extension, you need to apply to get your Developer ID enabled for KEXT development per the instructions on the Developer ID page.
Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"
Change history:
20 Jul 2016 — First published.
28 Mar 2019 — Added a link to the Apple PKI site. Other, minor changes.
15 Mar 2022 — Fixed the formatting. Updated the section number in the Developer ID CPS. Made other minor editorial changes.
Kernel
RSS for tagDevelop kernel-resident device drivers and kernel extensions using Kernel.
Posts under Kernel tag
48 Posts
Sort by:
Post
Replies
Boosts
Views
Activity
Hi.
I am facing a panic in distributed virtual filesystem of my own making.
The panic arises on attempt of copying a large folder, or writing a large file (both around 20gb).
An important note here is that the amount of files we try to copy is larger than available space (for testing purposes, the virtual file system had a capacity of 18 gigabytes).
The panic arises somewhere on 12-14gigabytes deep into copying. On the moment of panic, there are still several gigabytes of storage left.
The problem is present for sure for such architectures and macOS versions:
Sonoma 14.7.1 arm64e
Monterey 12.7.5 arm64e
Ventura 13.7.1 intel
Part from panic log from Ventura 13.7.1 intel, with symbolicated addresses:
panic(cpu 2 caller 0xffffff80191a191a): watchdog timeout: no checkins from watchdogd in 90 seconds (48 total checkins since monitoring last enabled)
Panicked task 0xffffff907c99f698: 191 threads: pid 0: kernel_task
Backtrace (CPU 2), panicked thread: 0xffffff86e359cb30, Frame : Return Address
0xffffffff001d7bb0 : 0xffffff8015e70c7d mach_kernel : _handle_debugger_trap + 0x4ad
0xffffffff001d7c00 : 0xffffff8015fc52e4 mach_kernel : _kdp_i386_trap + 0x114
0xffffffff001d7c40 : 0xffffff8015fb4df7 mach_kernel : _kernel_trap + 0x3b7
0xffffffff001d7c90 : 0xffffff8015e11971 mach_kernel : _return_from_trap + 0xc1
0xffffffff001d7cb0 : 0xffffff8015e70f5d mach_kernel : _DebuggerTrapWithState + 0x5d
0xffffffff001d7da0 : 0xffffff8015e70607 mach_kernel : _panic_trap_to_debugger + 0x1a7
0xffffffff001d7e00 : 0xffffff80165db9a3 mach_kernel : _panic_with_options + 0x89
0xffffffff001d7ef0 : 0xffffff80191a191a com.apple.driver.watchdog : IOWatchdog::userspacePanic(OSObject*, void*, IOExternalMethodArguments*) (.cold.1)
0xffffffff001d7f20 : 0xffffff80191a10a1 com.apple.driver.watchdog : IOWatchdog::checkWatchdog() + 0xd7
0xffffffff001d7f50 : 0xffffff80174f960b com.apple.driver.AppleSMC : SMCWatchDogTimer::watchdogThread() + 0xbb
0xffffffff001d7fa0 : 0xffffff8015e1119e mach_kernel : _call_continuation + 0x2e
Kernel Extensions in backtrace:
com.apple.driver.watchdog(1.0)[BD08CE2D-77F5-358C-8F0D-A570540A0BE7]@0xffffff801919f000->0xffffff80191a1fff
com.apple.driver.AppleSMC(3.1.9)[DD55DA6A-679A-3797-947C-0B50B7B5B659]@0xffffff80174e7000->0xffffff8017503fff
dependency: com.apple.driver.watchdog(1)[BD08CE2D-77F5-358C-8F0D-A570540A0BE7]@0xffffff801919f000->0xffffff80191a1fff
dependency: com.apple.iokit.IOACPIFamily(1.4)[D342E754-A422-3F44-BFFB-DEE93F6723BC]@0xffffff8018446000->0xffffff8018447fff
dependency: com.apple.iokit.IOPCIFamily(2.9)[481BF782-1F4B-3F54-A34A-CF12A822C40D]@0xffffff80188b6000->0xffffff80188e7fff
Process name corresponding to current thread (0xffffff86e359cb30): kernel_task
Boot args: keepsyms=1
Mac OS version:
22H221
Kernel version:
Darwin Kernel Version 22.6.0: Thu Sep 5 20:48:48 PDT 2024; root:xnu-8796.141.3.708.1~1/RELEASE_X86_64
The origin of the problem is surely inside my filesystem. However, the panic happens not there but somewhere in watchdog. As far as I can tell, the source code for watchdog is not available for public.
I can't understand what causes the panic.
Let's say we have run out of space. Couldn't write data. Writing received a proper error message and aborted. That's what is expected.
However, it is unclear for why the panic arises.
Hi.
I am developing a custom virtual file system and facing such behaviour:
Upon using some graphical apps, for example Adobe Media Encoder, attempting to navigate inside my filesystem deeper than root folder will fail - nothing will happen on "double click" on that subfolder. Another problem, is that whether I try to re-navigate into root directory, it will be empty.
The problem is not present for most GUI apps - for example navigation inside Finder, upon choosing download path for file in Safari, apps like Microsoft Word, Excel and other range of applications work totally correctly.
A quick note here. From what I have seen - all apps that work correctly actually have calls to VFS_VGET - a predefined vfs layer hook. Whether the Adobe Media Encoder does not call for it - neither in my filesystem, nor in Samba, so my guess is that some applications have different browsing and retrieving algorithm. Is there anything I should examine further ? Default routines (vnop_open, vnop_lookup, vnop_readdir, vnop_close) behave as expected, without any errors.
P.S. This application (Adobe Media Encoder) works properly on Samba.
I cannot find this specific KDK for my build 22H417. I need help locating and downloading this Developer Kit.
Error Domain=KMErrorDomain Code=34 "Missing Developer Kit: As of macOS 13.0, you will need to install a KDK matching your build 22H417 to rebuild kernel collections." UserInfo={NSLocalizedDescription=Missing Developer Kit: As of macOS 13.0, you will need to install a KDK matching your build 22H417 to rebuild kernel collections.}
I
SIGPIPE is an ongoing source of grief on Apple systems [1]. I’ve talked about it numerous times here on the forums. It cropped up again today, so I decided to collect my experiences into one post.
If you have questions or comments, please put them in a new thread. Put it in the App & System Services > Core OS topic area so that I see it.
Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"
[1] Well, on Unix-y systems in general, but my focus is Apple systems (-:
Debugging Broken Pipes
On Unix-y systems, writing to a pipe whose read side is closed will raise a SIGPIPE signal. The default disposition of that signal is to terminate your process [1]. Broken pipe terminations are tricky to debug on Apple systems because the termination doesn’t generate a crash report.
For example, consider this code:
let (read, write) = try FileDescriptor.pipe()
// This write works.
try write.writeAll("Hello Cruel World!".utf8)
let msg = try read.read(maxCount: 256)
… do something with `msg` …
// But if you close the read side…
try read.close()
// … the write call raises a `SIGPIPE`.
try write.writeAll("Goodbye Cruel World!".utf8)
Note This code relies on some extensions to FileDescriptor type that make it easier to call the pipe and write system calls. For more information about how I set that up, see Calling BSD Sockets from Swift.
If you put this in an iOS app and run it outside of Xcode, the app will terminate without generating a crash report.
This logic also applies to BSD Sockets. Writing to a disconnected socket may also trigger a SIGPIPE. This applies to the write system call and all the send variants: send, sendto, and sendmsg).
IMPORTANT Broken pipe terminations are even more troubling with sockets because sockets are commonly used for networking, where you have no control over the remote peer.
It’s easy to reproduce this signal with Unix domain sockets:
let (read, write) = try FileDescriptor.socketPair(AF_UNIX, SOCK_STREAM, 0)
// This write works.
try write.writeAll("Hello Cruel World!".utf8)
let msg = try read.read(maxCount: 256)
… do something with `msg` …
// But if you close the read side…
try read.close()
// … the write call raises a `SIGPIPE`.
try write.writeAll("Goodbye Cruel World!".utf8)
However, this isn’t limited to just Unix domain sockets; TCP sockets are a common source of broken pipe terminations.
[1] At first blush this API design might seem bananas, but it kinda makes sense in the context of traditional Unix command-line tools.
Confirm the Problem
The primary symptom of a broken pipe problem is that your app terminates without generating a crash report. Unfortunately, that’s not definitive. There are other circumstances where your app can terminate without generating a crash report. For example, another common cause of such terminations is the app calling exit.
There all two ways you can confirm this problem. The first relies on Xcode. Run your app in the Xcode debugger and, if it suddenly stops with the message Terminated due to signal 13, you know you’ve been terminated because of a broken pipe.
IMPORTANT Double check that the signal number is 13, the value of SIGPIPE.
If you can’t reproduce the problem in Xcode, look in the system log. When an app terminates the system records information about the reason. The exact log message varies from platform to platform, and from OS version to OS version. However, in the case of a SIGPIPE termination there’s usually a log entry containing PIPE or SIGPIPE, or that references signal 13.
For example, on iOS 18.2.1, I see this log entry:
type: default
time: 11:59:00.321882+0000
process: SpringBoard
subsystem: com.apple.runningboard
category: process
message: Firing exit handlers for 16876 with context <RBSProcessExitContext| specific, status:<RBSProcessExitStatus| domain:signal(2) code:SIGPIPE(13)>>
The log message contains both SIGPIPE and the SIGPIPE signal number, 13.
For more information about accessing the system log, see Your Friend the System Log.
Locate the Problem
Once you’ve confirmed that you have a broken pipe problem, you need to locate the source of it. That is, what code within your process is writing to a broken pipe?
If you can reproduce the problem in Xcode, configure LLDB to stop on SIGPIPE signals:
(lldb) process handle -s true SIGPIPE
NAME PASS STOP NOTIFY
=========== ===== ===== ======
SIGPIPE true true false
When the process writes to a broken pipe, Xcode stops in the debugger. Look at the backtrace in the Debug navigator to find the offending write.
If you can’t reproduce the problem in Xcode, one option is to add a signal handler that catches the SIGPIPE and triggers a crash. For example:
#include <signal.h>
static void sigpipeHandler(int sigNum) {
__builtin_trap();
}
extern void installSIGPIPEHandler(void) {
signal(SIGPIPE, sigpipeHandler);
}
Here the signal handler, sigpipeHandler, forces a crash by calling the __builtin_trap function.
IMPORTANT This code is in C, and uses __builtin_trap rather than abort, because of the very restricted environment in which the signal handler runs [1].
With this signal handler in place, writing to a broken pipe generates a crash report. Within that crash report, the crashing thread backtrace gives you a hint as to the location of the offending write. For example:
0 SIG-PIPETest … sigpipeHandler + 8
1 libsystem_platform.dylib … _sigtramp + 56
2 libswiftSystem.dylib … closure #1 in FileDescriptor._writeAll<A>(_:) + 100
3 libswiftSystem.dylib … partial apply for closure #1 in FileDescriptor._writeAll<A>(_:) + 20
4 libswiftSystem.dylib … partial apply for closure #1 in Sequence._withRawBufferPointer<A>(_:) + 108
5 libswiftCore.dylib … String.UTF8View.withContiguousStorageIfAvailable<A>(_:) + 108
6 libswiftCore.dylib … protocol witness for Sequence.withContiguousStorageIfAvailable<A>(_:) in conform…
7 libswiftCore.dylib … dispatch thunk of Sequence.withContiguousStorageIfAvailable<A>(_:) + 32
8 libswiftSystem.dylib … Sequence._withRawBufferPointer<A>(_:) + 472
9 libswiftSystem.dylib … FileDescriptor._writeAll<A>(_:) + 104
10 SIG-PIPETest … FileDescriptor.writeAll<A>(_:) + 28
…
Note The write system call is not shown in the backtrace. That’s because the crash reporter is not backtracing correctly across the signal handler stack frame that was inserted by the kernel between frames 1 and 2 [1]. Fortunately that doesn’t matter here, because we primarily care about our code, which is visible in frame 10.
I can’t see any problem with putting this code in your development build, or even deploying it to your beta testers. Think carefully before putting it in a production build that you deploy to all your users. Signal handlers are tricky [1].
[1] For all the gory details on that topic, see Implementing Your Own Crash Reporter for more information about that issue.
[2] This is one of the gory details covered by Implementing Your Own Crash Reporter.
Resolve the Problem
The best way to resolve this problem depends on whether it’s being caused by a pipe or a socket. The socket case is easy: Use the SO_NOSIGPIPE socket option to disable SIGPIPE on the socket. Once you do that, writing to the socket when it’s disconnected will return an EPIPE error rather than raising the SIGPIPE signal.
For example, you might tweak the code above like so:
let (read, write) = try FileDescriptor.socketPair(AF_UNIX, SOCK_STREAM, 0)
try read.setSocketOption(SOL_SOCKET, SO_NOSIGPIPE, 1 as CInt)
try write.setSocketOption(SOL_SOCKET, SO_NOSIGPIPE, 1 as CInt)
Note Again, this is using helpers from Calling BSD Sockets from Swift.
The situation with pipes is tricky. Apple systems have no way to disable SIGPIPE on a pipe, leaving you with two less-than-ideal options:
Disable SIGPIPE globally. To do this, call signal with SIG_IGN:
signal(SIGPIPE, SIG_IGN)
The downside to this approach is that affects the entire process. You can’t, for example, use this technique in library code.
Switch to Unix domain sockets. Rather than use a pipe for your IPC, use Unix domain sockets instead. As they’re both file descriptors, it’s usually quite straightforward to make this change.
The downside here is obvious: You need to modify your IPC code. That might be problematic, for example, if this IPC code is embedded in a framework that you don’t build from source.
I've having trouble deleting AppleDouble files residing on my custom filesystem through Finder.
This also affects files that use the AppleDouble naming convention, i.e. their names start with '._', but aren't AppleDoubles themselves.
dtrace output
In vnop_readdir, 'struct dent/dentry' is set up for dotbar files and written to the uio_t buffer.
It's just that my vnop_remove is never called for dotbar files, and I don't understand why not.
Dotbar files are removed successfully, when deleted through command line.
For SMBClients, vnop_readdir is followed by vnop_access, followed by vnop_lookup, followed by vnop_remove of dotbar files.
SMBClient rm dotbar files dtrace output
Implementing vnop_access for my filesystem did not result in the combination of vnop_lookup and vnop_remove being called for dotbar files.
Perusing the kernel sources, I observed the following functions that might be involved, but I have not way of verifying this, as none of the functions of interest are dtrace(1)-able, rmdir_remove_orphaned_appleDouble() in particular.
rmdir_remove_orphaned_appleDouble() -> VNOP_READDIR().
rmdirat_internal() -> rmdir_remove_orphaned_appleDouble()
unlinkat()-> rmdirat_internal()
rmdir()-> rmdirat_internal()
Any pointers on how dotbar files may be removed through Finder would be greatly appreciated.
Attempting to acquire the value of the 'kern.hostname' ctl from a kext by calling sysctlbyname() returns EPERM with no hostname returned.
sysctlbyname() is aliased to kernel_sysctlbyname():
config/Libkern.exports:839:_sysctlbyname:_kernel_sysctlbyname
Looking at the implementation of kernel_sysctlbyname(), EPERM is returned by sysctl_root(). Not sure how to correctly identify the point of failure.
Alternately, calling
sysctlbyname("hw.ncpu")
does return the value set for the ctl.
The kext was compiled with SYSCTL_DEF_ENABLED defined to have the relevant section of sys/sysctl.h enabled.
bsd_hostname() is a private symbol which is inaccessible to my kext.
% sysctl -n kern.hostname
does return the host name, so the ctl must be set.
Is it possible to get the name of a host from the context of my kext?
Thanks.
Firstly, I realise that Kexts are deprecated. And for my needs, user-space IOKit from an application might be more than I’ll ever need, but I can’t help ensuring I’ve got all my ducks in a row while I’m designing an app.
I noticed in the above Kext deprecation notice that it’s more about specific APIs that are deprecated rather than the kext mechanism itself (no mention of IOKit kernel APIs there). Along with the fact that a reboot would be required and various other policy changes.
While reading up on System Extensions and the various newer tools, DriverKit, Endpoint Security etc, I’ve noticed there’s no mention of this “IOKit Driver” target/deliverable in the Xcode template chooser. It looks like, from the requirements, that DriverKit is aimed at hardware manufacturers, with a need to request the DriverKit entitlement for development.
With respect to Apple’s internal Kext deprecation roadmap, how safe is it to build a product that relies on using an IOKit Driver and are there any requirements similar to DriverKit? Would a developer need to request an entitlement?
If I developed an app using user-space IOKit and, for illustration purposes let’s say I also included an IOKit Driver in the app bundle. If I used the IOKit API, including header file constants only (no digging around in the IORegistryExplorer for non-public keys), as Apple intends them to be used, would this fall foul of any App Store rules that anyone is aware of?
Put another way, would making use of an IOKit Driver disqualify my app from being distributed via the App Store, similar to an Endpoint Security extension?
As an aside, there are a lot of API in the ES Framework that could be used to build apps that have nothing to do with Endpoint Security. File system related apps for example. It’s a shame there isn’t an enhanced middle ground between FSEvents and Endpoint Security framework.
Is there a way I can use the lldb user defined commands, which I can load from a KDK, to determine how much memory my kext has consumed? Is there a way to figure out what size blocks they are?
I have already run the showallclasses command and the output has assured me that I am not leaking any objects which are defined by my driver's classes.
Thanks,
Tim
I saw that the vmmap program uses task-read_for_pid, and I want to use the task-read_for_pid export function to read process memory and check for any issues, but I don't have permission. How can I apply for it
Accessing a directory on my custom distributed filesystem results in a kernel panic.
According to the backtrace, the last function called before the panic is triggered is mac_label_verify().
See the backtrace file attached.
mac_label_verify-panic.txt
The panic manifests itself given the following conditions:
Machine-a: make a directory in Finder.
Machine-b: remove the directory created on machine-a in Finder.
Machine-a: access the directory removed on machine-b in Finder. Kernel panic ensues.
The panic is reproducible on both Apple Silicon and x86-64.
The backtrace is for x86-64 as I wasn't able to symbolicate it on Apple Silicon.
Not sure how to tackle this one.
Any pointers would be much appreciated.
Extracting an archive into the same directory on my custom filesystem more than once fails with the following message:
Unable to finish expanding 'misc.tar.xz' into 'extractme'.
Could not move 'misc' into destination directory.
I.e. initial extraction succeeds with archive contents extracted into extractme/misc.
Subsequent extraction fails to rename extractme.sb-db71cd27-lFjN1f/misc to extractme/misc 2.
This behaviour is observed on macOS Monterey and Ventura.
It does work as expected on macOS Sonoma though.
Dtrace(1)-ing the archive being extracted over smbfs results in the following sequence of calls being made:
2 -> smbfs_vnop_lookup AUHelperService-2163 -> extractme/misc 2 nameiop:0
2 <- smbfs_vnop_lookup AUHelperService-2163 -> extractme/misc 2 -> 2 ;ENOENT
2 -> smbfs_vnop_lookup AUHelperService-2163 -> extractme.sb-db71cd27-lFjN1f/misc nameiop:0x2 ;DELETE
2 <- smbfs_vnop_lookup AUHelperService-2163 -> extractme.sb-db71cd27-lFjN1f/misc -> 0
2 -> smbfs_vnop_lookup AUHelperService-2163 -> extractme/misc 2 nameiop:0x3 ;RENAME
2 <- smbfs_vnop_lookup AUHelperService-2163 -> extractme/misc 2 -> EJUSTRETURN
1 -> smbfs_vnop_rename AUHelperService-2163 -> extractme.sb-db71cd27-lFjN1f/misc -> extractme/nil
2 <- smbfs_vnop_rename AUHelperService-2163 -> extractme.sb-db71cd27-lFjN1f/misc -> extractme/nil -> 0
2 -> smbfs_vnop_lookup AUHelperService-2163 -> TheRooT/extractme/misc 2 nameiop:0
3 <- smbfs_vnop_lookup AUHelperService-2163 -> TheRooT/extractme/misc 2 -> 0 ;Successful lookup
What I don't understand is what causes vnop_lookup to be called for misc to be removed from the temporary directory and renamed into 'misc 2' and placed in the destination directory, 'extractme' via vnop_rename?
I had a look at smbfs_vnop_lookup and rename and didn't see anything that would cause 'misc 2' to come into being.
Based on the output of the dtrace(1) script running on my custom filesystem, there are no vnop_lookup and vnop_rename calls being made to remove the 'misc' directory from the temporary directory and to rename it to 'misc 2' and place it in the destination directory at extractme.
Archive extraction proceeds no further after extracting the archive contents into the temporary directory.
What am I missing?
I work on EdenFS, an open-source Virtual Filesystem that runs on macOS, Linux, and Windows. My team is very interested in using FSKit as the basis for EdenFS on macOS, but have found the documentation to be lacking and contains some mixed messaging on the future of FSKit. Below are a few questions that don’t seem to be fully covered by the current documentation:
Does FSKit support process attribution? Each FUSE request provides a requester Process ID (and other information) through the fuse_in_header structure. Does FSKit pass similar information along for each request?
Does the reclaimItem API function similarly to FUSE’s forget operation? If not, what are the differences? See #1 below for why forget/reclaimItem matters to us.
Is Apple committed to releasing and supporting FSKit? Is there any timeline for release that we can plan around?
Does FSKit have known performance/scalability limitations? We provide alternative methods that clients can use to make bulk requests to EdenFS, but some clients will necessarily be unable to use those and stress the default filesystem APIs. Throughput (on the order of tens of thousands of filesystem requests per minute) and request size are the main concerns, followed closely by directory size restrictions.
Why we’re interested in FSKit
As mentioned above, my team supports EdenFS on 3 platforms. On Linux, we utilize FUSE; on Windows, we utilize ProjectedFS; and on macOS, we’ve utilized a few different solutions in the past. We first utilized the macFUSE kext, which was great while it lasted. Due to (understandable) changes in supporting kernel extensions, we were forced to move to NFS version 3. NFS has been lackluster in comparison (and our initial investigations show that NFS version 4(.2) would be similar). We have had numerous scalability and reliability issues, some listed below:
NFS does not provide a forget API similar to FUSE. EdenFS is forced to remember all file handles that have been loaded because the kernel never informs us when all references to that file handle have been dropped. We can hackily infer that a file handle should never be referenced again in some cases, but a large number of file handles end up being remembered forever. Many of our algorithms scale with the number of file handles that Eden has to consider, and therefore performance issues are inevitable after some time.
NFS does not provide information about clients (requesters). We cannot tell which processes are sending EdenFS requests. This attribution is important due to issue #1. We are forced to work with tool owners to modify their applications to be VFS-friendly. If we can’t track down which tools are behaving poorly, they will continue to load excess file handles and cause performance issues.
NFS “Server connections interrupted:” dialog during heavy load. Under heavy load, either EdenFS or system-wide, our users experience this dialog pop-up and are confused as to how they should respond (Ignore or Disconnect All). They become blocked in their work, and will be further blocked if they click “Disconnect All” as that unmounts their EdenFS mount. This forces them to restart EdenFS or reboot their laptop to remediate the issue.
The above issues make us extremely motivated to use FSKit and partner with Apple to flesh out the final version of the FSKit API. Our use case likely mirrors what other user-space filesystems will be looking for in the FSKit API (albeit at a larger scale than most), and we’re willing to collaborate to work out any issues in the current FSKit offerings.
Since I haven't heard so much as a peep from Apple on this, I thought I'd take a poll here on how long I could expect an easily reproducible (albeit possibly obscure) kernel panic to be fixed.
I was under the impression that kernel panics were a big deal but it's been almost 2 months since I updated from macOS 14 to macOS 15.0 dev beta 7 / public beta 5 when I originally came across and reported a panic triggered while playing StarCraft II.
I've been able to consistently trigger panics playing certain (maybe all) Co-op maps in SC2 and since my first report Aug 22, I've filed 8 additional bug reports, each automatically generated after hitting yet another panic. (I'm not sure exactly who is able to view these but for what it's worth, these are the reports I've filed so far: FB14886510, FB14905773, FB14960435, FB15304609, FB15391195, FB15467943, FB15468127, FB15491485, FB15491684.)
A few other people have reported the issue to SC2's developer, Blizzard, and apparently Blizzard has acknowledged they're aware of the problem so it's safe to rule out the possibility of a hardware defect or other issue specific only to my computer.
The logs point the blame at the AppleDCP driver, although I suppose the problem could technically be in the DCP firmware instead. Regardless, Apple's code is clearly at fault here.
I'll admit the importance of a video game isn't exactly like keeping the power on at a hospital but I don't know why it would be deemed particularly unimportant either.
At 53 days in, am I wrong to expect this to have been fixed by now or is Apple really being that slow?
I’ve talked about EINTR a bunch of times here on DevForums. Today I found myself talking about it again. On reading my other explanations, I didn’t think any of them were good enough to link to, so I decided to write it up properly.
If you have questions or comments, please put them in a new thread here on DevForums. Use the App & System Services > Core OS topic area so that I see it.
Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"
Understanding EINTR
Many BSD-layer routines can fail with EINTR. To see this in action, consider the following program:
import Darwin
func main() {
print("will read, pid: \(getpid())")
var buf = [UInt8](repeating: 0, count: 1024)
let bytesRead = read(STDIN_FILENO, &buf, buf.count)
if bytesRead < 0 {
let err = errno
print("did not read, err: \(err)")
} else {
print("did read, count: \(bytesRead)")
}
}
main()
It reads some bytes from stdin and prints the result. Build this and run it in one Terminal window:
% ./EINTRTest
will read, pid: 13494
Then, in other window, stop and start the process by sending it the SIGSTOP and SIGCONT signals:
% kill -STOP 13494
% kill -CONT 13494
In the original window you’ll see something like this:
% ./EINTRTest
will read, pid: 13494
zsh: suspended (signal) ./EINTRTest
%
did not read, err: 4
[1] + done ./EINTRTest
When you send the SIGSTOP the process stops and the shell tells you that. But looks what happens when you continue the process. The read(…) call fails with error 4, that is, EINTR. The read man page explains this as:
[EINTR] A read from a slow device was interrupted before any data arrived by the delivery of a signal.
That’s true but unhelpful. You really want to know why this error happens and what you can do about it. There are other man pages that cover this topic in more detail — and you’ll find lots of info about it on the wider Internet — but the goal of this post is to bring that all together into one place.
Signal and Interrupts
In the beginning, Unix didn’t have threads. It implemented asynchronous event handling using signals. For more about signals, see the signal man page.
The mechanism used to actually deliver a signal is highly dependent on the specific Unix implementation, but the general idea is that:
The system decides on a specific process (or, nowadays, a thread) to run the signal handler.
If that’s blocked inside the kernel waiting for a system call to complete [1], the system unblocks the system call by failing it with an EINTR error.
Thus, every system call that can block [2] might fail with an EINTR. You see this listed as a potential error in the man pages for read, write, usleep, waitpid, and many others.
[1] There’s some subtlety around the definition of system call. On traditional Unix systems, executables would make system calls directly. On Apple platforms that’s not supported. Rather, an executable calls a routine in the System framework which then makes the system call. In this context the term system call is a shortcut for a System framework routine that maps to a traditional Unix system call.
[2] There’s also some subtlety around the definition of block. Pretty much every system call can block for some reason or another. In this context, however, a block means to enter an interruptible wait state, typically while waiting for I/O. This is what the above man page quote is getting at when it says slow device.
Solutions
This is an obvious pitfall and it would be nice if we could just get rid of it. However, that’s not possible due to compatibility concerns. And while there are a variety of mechanism to automatically retry a system call after a signal interrupt, none of them are universally applicable. If you’re working on a large scale program, like an app for Apple’s platforms, you only good option is to add code to retry any system call that can fail with EINTR.
For example, to fix the program at the top of this post you might wrap the read(…) system call like so:
func readQ(_ d: Int32, _ buf: UnsafeMutableRawPointer!, _ nbyte: Int) -> Int {
repeat {
let bytesRead = read(d, buf, nbyte)
if bytesRead < 0 && errno == EINTR { continue }
return bytesRead
} while true
}
Note In this specific case you’d be better off using the read(into:retryOnInterrupt:) method from System framework. It retries by default (if that’s not appropriate, pass false to the retryOnInterrupt parameter).
You can even implement the retry in a generic way. See the errnoQ(…) snippet in QSocket: System Additions.
Library Code
If you’re writing library code, it’s important that you handle EINTR so that your clients don’t have to. In some cases it might make sense to export a control for this, like the retryOnInterrupt parameter shown in the previous section, but it should default to retrying.
If you’re using library code, you can reasonably expect it to handle EINTR for you. If it doesn’t, raise that issue with the library author. And you get this error back from an Apple framework, like Foundation or Network framework, please file a bug against the framework.
Is there an option to create a macOS filter driver for the IONVMeFamily Driver that would enable sending NVMe Admin commands( likes security send , security receive, Firmware download)? This approach would allow us to reuse the IO path provided by the family driver.
Hello,
It seems like the Kernel Debug Kit for macOS 15.0.1 (24A348) is missing from the list of downloads at developer.apple.com. It would be great if you could add them to the list of available downloads.
When trying to rebuild the kernel it fails with the following error message:
Error Domain=KMErrorDomain Code=34 "Missing Developer Kit: As of macOS 13.0, you will need to install a KDK matching your build 24A348 to rebuild kernel collections." UserInfo={NSLocalizedDescription=Missing Developer Kit: As of macOS 13.0, you will need to install a KDK matching your build 24A348 to rebuild kernel collections.}
But my macOS version is 15.0.1. Is there a workaround for this?
So if one were to start the attempt of porting an existing kext VFS filesystem, to use the new FSKit (Since presumably kexts could go away), how would that look now?
Is it ready? Are there any samples out there that already works (Filesystem using FSKit) ?
How is the documentation? ChatGPT did not seem to know much at all.
What would be Apple's reception to that?
How flexible is FSKit ? Is it locked to the idea of a mount is connected to a physical device (or partition)? Or is it more virtual, in that I will have a pool of disks, and present 1, or many, mount points?
I'm observing all sorts of race conditions occurring in various VNOPs my custom filesystem implements.
I'm inclined to attribute this to my implementation not following the locking rules expected by the system of a 3rd party filesystem as well as it should.
I've looked at how locking is done in Apple's own implementation of Samba and NFS clients.
The Samba client uses read/write locks to protect its node from data races.
While the NFS client uses mutex locks for the same purpose.
I realised that I don't have a clear model in my head of how locking should be done properly.
Thus my question, what are the locking rules for VFS and VNOP operations?
Thanks.
I am porting a working kernel extension IOKit driver to a DriverKit system extension. Our device is a PCI device accessed through Thunderbolt. The change from IOPCIFamily to PCIDriverKit has some differences in approach, though.
Namely, in IOKit / IOPCIFamily, this was the correct way to become Bus Leader:
mPCIDevice->setBusLeadEnable(true); // setBusMasterEnable(..) deprecated in OS 12.4
but now, PCIDriverKit's IOPCIDevice does not have that function. Instead I am doing the following:
// Set Bus Leader and Memory Space enable
uint16_t commandRegister = 0;
ivars->mPCIDevice->ConfigurationRead16(kIOPCIConfigurationOffsetCommand, &commandRegister);
commandRegister |= (kIOPCICommandBusLead | kIOPCICommandMemorySpace);
ivars->mPCIDevice->ConfigurationWrite16(kIOPCIConfigurationOffsetCommand, commandRegister);
But I am not convinced this is working (I am still experiencing unexpected errors when attempting to DMA from our device, using the same steps that work for the kernel extension).
The only hint I can find in the online documentation is here, which reads:
Note
The endpoint driver is responsible for enabling the Memory Space Enable and Bus Master Enable settings each time it configures the PCI device. When a crash occurs, or when the system unloads your driver, the system disables these features.
...but that does not state directly how to enable bus leader status. What is the "PCIDriverKit approved" way to become bus leader?
Is there a way to verify/confirm that a device is bus leader? (This would be helpful to prove that bus leadership is not the issue for DMA errors, as well as to confirm that bus leadership was granted).
Thanks in advance!
Hello all,
Recently I observed a strange behaviour on macOS.
Some apps with UI, after you quit them (right click on the Dock, select Quit or select Quit from the menubar), the apps are not actually quitting immediately, but in a few seconds (including in Activity Monitor the apps are staying alive). Also, if you open the apps again fast, the same PID is kept.
Not all apps do this, some of them, for example WhatsApp.
I'm not referring to closing all windows, but explicitly quitting.
This was not the case in the past. Is there any reason for this? Is some kind of optimisation I'm not aware of?
The actual issue is that in a Swift developed app events like NSWorkspace.didLaunchApplicationNotification
or NSWorkspace.didTerminateApplicationNotification are not triggered.
Is there any way to tell if an app was closed, even if macOS still keeps it around for a few more seconds?
Thank you.