# IOUserSCSIParallelInterfaceController: `UserGetDataBuffer` returns a zero-filled buffer for some writes

Is UserGetDataBuffer the right way for a software (no-DMA) controller to get at write payload on the CPU at all?

This is a grey area. Strictly speaking, I don't think virtual devices were ever really factored into the design because, quite frankly, virtual controllers would have been WAY easier to implement at other points in the storage stack. Having said that, this is also the only way I can see to make this work at all within DriverKit. And...

The docs frame it as the exception and otherwise point at fBufferIOVMAddr, but that reads like an IOVM/physical segment I don't think I can map for CPU access — is there a CPU-accessible path I'm missing?

...no, you're not missing anything.

I got stuck, so I disassembled IOSCSIParallelFamily to try to understand where the buffer comes from. In UserGetDataBuffer_Impl it looks like it allocates a fresh IOBufferMemoryDescriptor, zero-fills it, and only copies the client data in when the transfer direction is "from initiator to target".

Yes, that's basically what it does.

The direction it tests there is read from the SCSITask itself (the byte at SCSITask+0x5b, via GetDataTransferDirection), not from the fTransferDirection field I get in the task struct.

No, that's not quite what's happening. It's actually finding the right task by calling IOSCSIParallelInterfaceController::FindTaskForControllerIdentifier() with fControllerTaskIdentifier as the task ID. That leads to:

Is there something I should be doing at task setup / UserMapHBAData / completion so this direction isn't stale when a task object is recycled?

Are you returning a unique ID in UserMapHBAData? If you're not, then yes, bad things will happen, as we'll end up retrieving the wrong SCSITask for UserGetDataBuffer - so not only will you get zero write, I think you'll actually end up writing entirely the wrong data.

Note that this requirement is mentioned in the class reference:

The framework calls this method for every SCSIParallelTask immediately after creating the task in the kernel. The driver extension class should also set a unique task ID. The framework uses this ID to uniquely identify the corresponding SCSIParallelTask in the kernel.

Or is the SCSITask direction meant to always agree with fTransferDirection by the time my task-processing method runs, and a mismatch means I've done something wrong on my end?

I haven't done a full audit, but the code is straightforward enough that I don't see how we'd call with the "wrong" SCSITaskData. However, if UserMapHBAData didn't return a unique ID, then BAD things are going to happen, as we'll end up retrieving the wrong SCSITask object in UserMapHBAData. That basically matches what you're describing.

That leads to my question here:

This works almost all the time, but I've got an intermittent case I can't explain: for some WRITE tasks the buffer I get back is entirely zero-filled, even though it's a genuine write that should carry data.

How are you generating I/O? Is this just simple rdev node read/writes, or is there a live filesystem involved? Duplicate IDs might (sort of [1]) work for direct I/O, but they'd corrupt and possibly panic a real file system.

[1] You'd be reading/writing the wrong data, but that doesn't matter if nothing is actually interpreting that data.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Thanks Kevin. I went through your answer point by point with experiments — here's what I found.

it's not necessarily true that pwrite DID return EIO (at least for that log message)

I wasn't sure either, so I traced it with dtruss — in my case it really is pwrite(2) returning EIO (fd 3 = the raw device):

pwrite(0x3, ..., 0x1000, 0x2000)  = 4096 0
pwrite(0x3, "x\0", 0x1000, 0x0)   = -1 Err#5    ← the superblock write
write_nocancel(0x2, "nx_format:308: failed to write superblock to block 0: 5 - Input/output error\n", 0x4D)

lldb shows the failing buffer is a 32-byte-aligned heap pointer, so whether it straddles a page boundary is an allocator lottery — which also explains why diskutil eraseDisk only fails some of the time.

the problem is that you haven't properly configured all of the keys required by UserReportHBAConstraints

All seven keys are defined (and also republished as node properties via SetProperties, per FB21256805):

That satisfies maxTransferSize >= SegmentCount × SegmentByteCount (16384 = 1 × 16384), and the failing transfer is only 4 KiB.

I would strongly recommend writing your own test tool that generates arbitrarily sized I/O across a broad size range

Did that — byte-verified matrix over buffer alignments × sizes from 512 B to 128 MiB. Everything passes except one shape. Same device, same offset, same 4 KiB size, only the buffer placement varies:

• buffer within one 16 KiB page → OK (any alignment)
• block-aligned buffer straddling a page (e.g. page+0x3000, 8 KiB) → OK, split at the crossing
• sub-block-aligned buffer straddling a page (e.g. page+0x3800, 4 KiB) → EIO, deterministic

From the IOStorageFamily source this looks like IOBreaker::getBreakSize() truncating the split point to a device-block multiple and getting 0 for that shape, so the request fails with kIOReturnDMAError before anything reaches my dext — but I may well be misreading it.

So, two questions:

1. Am I doing something wrong in my configuration, or misreading how these buffers are supposed to be handled?
2. If not — what should happen to a sub-block-aligned, page-straddling buffer, and whose job is it to make it deliverable?

The behavior reproduces with a similar configuration:

FYI, your reply above was posted to the wrong thread. I've posted my reply on the original thread, here.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware