IOUserSCSIParallelInterfaceController: Single-Segment Requirement Breaks No-DMA / No-IOMMU Controllers

Question

Created 7h

Replies 1

Boosts 0

Participants 2

Summary

I have a userspace IOUserSCSIParallelInterfaceController dext presenting a virtual disk. It's a software / no-DMA controller — it moves all data with a kernel CPU copy via UserGetDataBuffer; it never does hardware DMA and never reads fBufferIOVMAddr. Buffered I/O works and the disk mounts. The problem is unbuffered / raw-device I/O.

For a data-carrying task, ProcessParallelTask calls PrepareForDMA then GenerateIOVMSegments on the task's IODMACommand, and if the segment count != 1 it fails the task with EIO before ever calling UserProcessParallelTask / UserProcessBundledParallelTasks — so UserGetDataBuffer never gets a chance. I already use UserGetDataBuffer and it works for single-segment (aligned) tasks, but a client buffer that straddles a page boundary produces 2 segments and is rejected at that gate.

Concrete symptom

newfs_apfs writing the container superblock to the raw device from a page-straddling malloc'd buffer:

nx_format:308: failed to write superblock to block 0: 5 - Input/output error

Minimal repro against my raw device: an aligned 4 KiB pwrite succeeds; a 4 KiB pwrite from a buffer straddling a 16 KiB page returns errno 5.

What I think is going on (not sure)

My guess is that the framework expects a DART/IOMMU to coalesce a scattered buffer into one IOVM segment, and my virtual controller doesn't have one (ioreg shows no mapper and no iommu-parent on the node — attached), so GenerateIOVMSegments emits raw physical segments and a straddling buffer stays 2 segments. But I don't know if that's actually the reason, or whether a no-DMA controller is even supposed to go through GenerateIOVMSegments at all — hence the questions below.

Questions

For a no-DMA controller that services data via UserGetDataBuffer, is there a supported way to make the framework deliver a task whose client buffer maps to more than one IOVM segment (relax/skip the GenerateIOVMSegments single-segment gate), so my upcall can run? UserGetDataBuffer returns a fresh contiguous buffer, so the original buffer's segment count shouldn't matter for a CPU-copy controller.
Is there a characteristic to declare a PIO / no-DMA / software controller so the framework skips segment generation for it?
Can a DriverKit controller get macOS to interpose an IOMapper/DART in front of it (so the buffer is coalesced into one segment) — via a property, a matching personality, or an intermediate provider nub? Or is a hardwareless controller simply not expected to support unbuffered/raw-device I/O?

Environment

Build: 26A5368g, arm64e
Full ioreg -w0 -r -c IOUserSCSIParallelInterfaceController attached.

Happy to file a Feedback with a sysdiagnose and the minimal repro.

Answer 1

DTS Engineer OP

Apple

50m

So, let me start here:

For a data-carrying task, ProcessParallelTask calls PrepareForDMA then GenerateIOVMSegments on the task's IODMACommand, and if the segment count != 1 it fails the task with EIO before ever calling UserProcessParallelTask / UserProcessBundledParallelTasks — so UserGetDataBuffer never gets a chance. I already use UserGetDataBuffer and it works for single-segment (aligned) tasks, but a client buffer that straddles a page boundary produces 2 segments and is rejected at that gate.

This is not what's going on, as PrepareForDMA will never[1] return more than one segment. There's a bit more detail in this post, but the loose summary is that the DART will happily do mappings FAR in excess of anything you'd ever practically do. Note that the design of IOUserSCSIParallelInterfaceController actually relies on this behavior, as it's why SCSIUserParallelTask only passes in a single address through fBufferIOVMAddr instead of passing in a segment list. It's relying on the DART to map an arbitrarily large buffer to a single range instead of using individual segments.

Next, a quick comment on this point:

Or is a hardwareless controller simply not expected to support unbuffered/raw-device I/O?

As far as the IOKit stack is concerned, there isn't really any difference between buffered and unbuffered I/O. They arrive through slightly different entry points (the VFS layer vs. IOBSDMediaClient), but after that point, the I/O paths are EXACTLY the same. It's critical to understand that because what DOES change here is that it can be easier to generate "arbitrarily" sized I/O through the raw path.

Don't ignore ANY failure here. The key word there is "easier"- that is, any failure in the raw I/O system is a potential buffered I/O failure that's simply harder to generate in real-world use.

Related to that point:

newfs_apfs writing the container superblock to the raw device from a page-straddling malloc'd buffer:

...I would strongly recommend writing your own test tool that generates arbitrarily sized I/O across a broad size range, probably all the way to 100s of MiB and even GiB. The system is designed to be able to divide ANY I/O request into a size you can handle, so "everything" should work.

In addition, at this early stage, every intermediate layer creates additional complication and confusion. Case in point, I wasted a considerable amount of time trying to figure out why pwrite returned EIO before I realized that it's not necessarily true that pwrite DID return EIO (at least for that log message). APFS has its own media client format, which means you're going down a completely different I/O path there.

In any case, shifting to pwrite specifically:

Minimal repro against my raw device: an aligned 4 KiB pwrite succeeds; a 4 KiB pwrite from a buffer straddling a 16 KiB page returns errno 5.

I don't know exactly what's gone wrong, but if I had to guess, the problem is that you haven't properly configured all of the keys required by UserReportHBAConstraints. There's a thread about this issue here, but the short summary is that improper or invalid configuration can produce exactly the failure you're seeing. In particular, I'd highlight this post, which describes how misconfiguration can mean that requests to large for your controller can reach the controller layer, at which point the controller layer will fail the request because your DEXT said it couldn't handle them:

(Referencing maxTransferSize returned through UserGetDMASpecification)

...I'm fairly convinced that maxTransferSize MUST either:

maxTransferSize >= kIOMaximumSegmentCountReadKey * kIOMaximumSegmentByteCountReadKey
OR
maxTransferSize >= kIOMaximumSegmentCountWriteKey * kIOMaximumSegmentByteCountWriteKey

...whichever of the two is larger. maxTransferSize basically defines "the largest possible transfer your controller could EVER handle", which would obviously be your segment count * segment size. Critically, using a smaller maxTransferSize won't cause an immediate failure, only a later failure if/when the kernel actually tried to "give you" a large enough transfer.

Putting that another way, SCSIControllerDriverKit guarantees your DEXT will never receive a transfer larger than maxTransferSize. It enforces that guarantee... by preemptively failing ANY transfer larger than "maxTransferSize". Your job is to then use the configuration keys defined by UserReportHBAConstraints to ensure that the storage system breaks up requests properly before they reach SCSIControllerDriverKit's final "check".

[1] While it is theoretically possible that PrepareForDMA could return multiple segments, I don't know how this would be done, I'm not sure it's actually possible within DriverKit, and most of our DEXTs specifically hard code that they will only EVER have one segment.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware