[DEXT Migration Issue] IOUserSCSIParallelInterfaceController fails to handle low-level I/O from `diskutil`

Question

charles.cc OP

Created 13h

Replies 1

Boosts 0

Participants 2

Hello everyone,

We are migrating our KEXT for a Thunderbolt storage device to a DEXT based on IOUserSCSIParallelInterfaceController.

We've run into a fundamental issue where the driver's behavior splits based on the I/O source: high-level I/O from the file system (e.g., Finder, cp) is mostly functional (with a minor ls -al sorting issue for Traditional Chinese filenames), while low-level I/O directly to the block device (e.g., diskutil) fails or acts unreliably. Basic read/write with dd appears to be mostly functional.

We suspect that our DEXT is failing to correctly register its full device "personality" with the I/O Kit framework, unlike its KEXT counterpart. As a result, low-level I/O requests with special attributes (like cache synchronization) sent by diskutil are not being handled correctly by the IOUserSCSIParallelInterfaceController framework of our DEXT.

Actions Performed & Relevant Logs

1. Discrepancy: diskutil info Shows Different Device Identities for DEXT vs. KEXT

For the exact same hardware, the KEXT and DEXT are identified by the system as two different protocols.

KEXT Environment:

   Device Identifier:         disk5
   Protocol:                  Fibre Channel Interface
   ...
   Disk Size:                 66.0 TB
   Device Block Size:         512 Bytes

DEXT Environment:

   Device Identifier:         disk5
   Protocol:                  SCSI
   SCSI Domain ID:            2
   SCSI Target ID:            0
   ...
   Disk Size:                 66.0 TB
   Device Block Size:         512 Bytes

2. Divergent I/O Behavior: Partial Success with Finder/cp vs. Failure with diskutil

High-Level I/O (Partially Successful): In the DEXT environment, if we operate on an existing volume (e.g., /Volumes/GammaCarry), file copy operations using Finder or cp succeed. Furthermore, the logs we've placed in our single I/O entry point, UserProcessParallelTask_Impl, are triggered.
- Side Effect: However, running ls -al on such a volume shows an incorrect sorting order for files with Traditional Chinese names (they appear before . and ..).
Low-Level I/O (Contradictory Behavior): In the DEXT environment, when we operate directly on the raw block device (/dev/disk5):
- diskutil partitionDisk ... -> Fails 100% of the time with the error: Error: -69825: Wiping volume data to prevent future accidental probing failed.
- dd command -> Basic read/write operations appear to work correctly (a write can be immediately followed by a read within the same DEXT session, and the data is correct).

3. Evidence of Cache Synchronization Failure (Non-deterministic Behavior)

The success of the dd command is not deterministic. Cross-environment tests prove that its write operations are unreliable:

First Test:
1. In the DEXT environment, write a file with random data to /dev/disk5 using dd.
2. Reboot into the KEXT environment.
3. Read the data back from /dev/disk5 using dd. The result is a file filled with all zeros.
- Conclusion: The write operation only went to the hardware cache, and the data was lost upon reboot.
Second Test:
1. In the DEXT environment, write the same random file to /dev/disk5 using dd.
2. Key Variable: Immediately after, still within the DEXT environment, read the data back once for verification. The content is correct!
3. Reboot into the KEXT environment.
4. Read the data back from /dev/disk5. This time, the content is correct!
- Conclusion: The additional read operation in the second test unintentionally triggered a hardware cache flush. This proves that the dd (in our DEXT) write operation by itself does not guarantee synchronization, making its behavior unreliable.

Our Problem

Based on the observations above, we have the conclusion:

High-Level Path (triggered by Finder/cp): When an I/O request originates from the high-level file system, the framework seems to enter a fully-featured mode. In this mode, all SCSI commands, including READ/WRITE, INQUIRY, and SYNCHRONIZE CACHE, are correctly packaged and dispatched to our UserProcessParallelTask_Impl entry point. Therefore, Finder operations are mostly functional.
Low-Level Path (triggered by dd/diskutil): When an I/O request originates from the low-level raw block device layer:
- The most basic READ/WRITE commands can be dispatched (which is why dd appears to work).
- However, critical management commands, such as INQUIRY and SYNCHRONIZE CACHE, are not being correctly dispatched or handled. This leads to the incorrect device identification in diskutil info and the failure of diskutil partitionDisk due to its inability to confirm cache synchronization.

We would greatly appreciate any guidance, suggestions, or insights on how to resolve this discrepancy. Specifically, what is the recommended approach within DriverKit to ensure that a DEXT based on IOUserSCSIParallelInterfaceController can properly declare its capabilities and handle both high-level and low-level I/O requests uniformly?

Thank you.

Charles

Boost

Answer 1

DTS Engineer OP

Apple

5h

So, let me start with a clarification here:

In the DEXT environment, when we operate directly on the raw block device (/dev/disk5):

The "raw" block device here would be "/dev/rdisk5", not "/dev/disk5". Your actually interacting with the "cached" device, not the "raw" device.

That leads to here:

First Test:

In the DEXT environment, write a file with random data to /dev/disk5 using dd.
Reboot into the KEXT environment.
Read the data back from /dev/disk5 using dd. The result is a file filled with all zeros.

Conclusion: The write operation only went to the hardware cache, and the data was lost upon reboot.

What actually happened in your DEXT for #1? Are you sure you received ANY I/O command? Also, have you tried this test entirely in the KEXT environment? In particular, what happens if you pull power from the machine (instead of cleanly shutting down)?

On KEXT side, I think what's actually happening here is the following:

The write hits the UBC (Universal Buffer Cache) without actually generating an I/O commands.
The shutdown process flushes everything out of the cache and the KEXT ensures everything is "finalized" to disk before it allows terminations.

I'd expect this to generally work the same in a DEXT, but if you're DEXT isn't ensuring everything is flushed during teardown, then that could cause data loss.

Second Test:

In the DEXT environment, write the same random file to /dev/disk5 using dd.
Key Variable: Immediately after, still within the DEXT environment, read the data back once for verification. The content is correct!

I'd be curious to see how the command stream differed here, but my guess about what happened here is that the system had to commit the write to disk before it triggered the read. However, it's also possible that the UBC returned the data you'd written directly from the cache, while also issuing a "flush" on it's own.

One thing I'd suggest testing here is exactly how your DEXT behaves if you run "diskutil eject". I think that should behave similar to what happens at shutdown, without the headache of trying to log/debug across reboots.

However, that leads to here:

diskutil partitionDisk ... -> Fails 100% of the time with the error: Error: -69825: Wiping volume data to prevent future accidental probing failed

Something different is going on here. In general, diskutil use the rdisk node, which means you're getting direct I/O requests without the write cache being involved.

The error code itself basically means "wiping failed", but that does narrow the scope enough that I'm sure this targeted the rdisk, not the disk. Do any writes make it your disk when you get this error?

Our code uses that error in two places, once for errors coming out of "pwrite" and the other if something goes wrong when setting up the initial buffers that are used the I/O source.

In terms of that second path, take a look at the code in IOMediaBSDClient.cpp, particularly dkioctl. IOMediaBSDClient is where things like ioctl() calls are "converted" into IOKit's architecture and, as you'll see, most of them are just reading properties off their underlying IOMedia object, all of which were propagated "up" from your controller. If you're not getting any writes, then I would start by comparing the IOMedia configuration of KEXT vs. DEXT and see if you can find anything that's wrong.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

0