I'm writing a software IOUserSCSIParallelInterfaceController in DriverKit — there's
no DMA hardware behind it (it forwards commands elsewhere), so for each command I read
the payload on the CPU by calling UserGetDataBuffer() from within my task-processing
method and then CreateMapping() on the returned IOBufferMemoryDescriptor.
This works almost all the time, but I've got an intermittent case I can't explain: for
some WRITE tasks the buffer I get back is entirely zero-filled, even though it's a
genuine write that should carry data. In those cases the
SCSIUserParallelTask.fTransferDirection I'm handed is
kSCSIDataTransfer_FromInitiatorToTarget, so the task itself looks like a perfectly
normal write to me. It seems to happen when the same task object gets reused — a read on
that task, then a write.
I got stuck, so I disassembled IOSCSIParallelFamily to try to understand where the
buffer comes from. In UserGetDataBuffer_Impl it looks like it allocates a fresh
IOBufferMemoryDescriptor, zero-fills it, and only copies the client data in when the
transfer direction is "from initiator to target". Roughly what I think I'm seeing:
; allocate a fresh IOBufferMemoryDescriptor, then bzero it
bl GetDataTransferDirection ; SCSIParallelTask -> SCSITask (+0x100), then ldrb w0, [x0, #0x5b]
cmp w0, #1 ; kSCSIDataTransfer_FromInitiatorToTarget ?
b.ne Lskip ; if not a write, leave the buffer zeroed
... client->readBytes(0, bounce, len) ; copy client -> bounce
Lskip:
... return bounce ; hand back the (possibly still-zeroed) buffer
The direction it tests there is read from the SCSITask itself (the byte at
SCSITask+0x5b, via GetDataTransferDirection), not from the fTransferDirection field
I get in the task struct. And in the failing cases that byte still seems to hold the
previous direction for that task (a read, 0x02), so the cmp w0, #1 doesn't match and
the copy is skipped — which would explain the zero buffer.
I could easily be misreading the disassembly, so I'd really appreciate a sanity check on the intended contract:
- Is
UserGetDataBufferthe right way for a software (no-DMA) controller to get at write payload on the CPU at all? The docs frame it as the exception and otherwise point atfBufferIOVMAddr, but that reads like an IOVM/physical segment I don't think I can map for CPU access — is there a CPU-accessible path I'm missing? - Is it my responsibility (or the layer above me) to make sure the task's data-transfer
direction is established before
UserGetDataBufferruns? Is there something I should be doing at task setup /UserMapHBAData/ completion so this direction isn't stale when a task object is recycled? - Or is the SCSITask direction meant to always agree with
fTransferDirectionby the time my task-processing method runs, and a mismatch means I've done something wrong on my end?
Any pointers on the intended behavior here would be a big help — thanks.
Is UserGetDataBuffer the right way for a software (no-DMA) controller to get at write payload on the CPU at all?
This is a grey area. Strictly speaking, I don't think virtual devices were ever really factored into the design because, quite frankly, virtual controllers would have been WAY easier to implement at other points in the storage stack. Having said that, this is also the only way I can see to make this work at all within DriverKit. And...
The docs frame it as the exception and otherwise point at fBufferIOVMAddr, but that reads like an IOVM/physical segment I don't think I can map for CPU access — is there a CPU-accessible path I'm missing?
...no, you're not missing anything.
I got stuck, so I disassembled IOSCSIParallelFamily to try to understand where the buffer comes from. In UserGetDataBuffer_Impl it looks like it allocates a fresh IOBufferMemoryDescriptor, zero-fills it, and only copies the client data in when the transfer direction is "from initiator to target".
Yes, that's basically what it does.
The direction it tests there is read from the SCSITask itself (the byte at SCSITask+0x5b, via GetDataTransferDirection), not from the fTransferDirection field I get in the task struct.
No, that's not quite what's happening. It's actually finding the right task by calling IOSCSIParallelInterfaceController::FindTaskForControllerIdentifier() with fControllerTaskIdentifier as the task ID. That leads to:
Is there something I should be doing at task setup / UserMapHBAData / completion so this direction isn't stale when a task object is recycled?
Are you returning a unique ID in UserMapHBAData? If you're not, then yes, bad things will happen, as we'll end up retrieving the wrong SCSITask for UserGetDataBuffer - so not only will you get zero write, I think you'll actually end up writing entirely the wrong data.
Note that this requirement is mentioned in the class reference:
The framework calls this method for every SCSIParallelTask immediately after creating the task in the kernel. The driver extension class should also set a unique task ID. The framework uses this ID to uniquely identify the corresponding SCSIParallelTask in the kernel.
Or is the SCSITask direction meant to always agree with fTransferDirection by the time my task-processing method runs, and a mismatch means I've done something wrong on my end?
I haven't done a full audit, but the code is straightforward enough that I don't see how we'd call with the "wrong" SCSITaskData. However, if UserMapHBAData didn't return a unique ID, then BAD things are going to happen, as we'll end up retrieving the wrong SCSITask object in UserMapHBAData. That basically matches what you're describing.
That leads to my question here:
This works almost all the time, but I've got an intermittent case I can't explain: for some WRITE tasks the buffer I get back is entirely zero-filled, even though it's a genuine write that should carry data.
How are you generating I/O? Is this just simple rdev node read/writes, or is there a live filesystem involved? Duplicate IDs might (sort of [1]) work for direct I/O, but they'd corrupt and possibly panic a real file system.
[1] You'd be reading/writing the wrong data, but that doesn't matter if nothing is actually interpreting that data.
__
Kevin Elliott
DTS Engineer, CoreOS/Hardware
