Should UserSendCBD work on UAS interfaces?

The device I am trying to develop a firmware updater for is an NVMe drive with a USB4 interface. It can connect in USB4 mode (tunneled NVMe), in USB 3 mode or in USB 2 mode.

In USB 2 and USB 3 mode, the device descriptor shows one interface with two alternates. Alternate 0 uses the bulk-only protocol, with one IN and one OUT pipe. Alternate 1 uses the UAS protocol, with two IN and two OUT pipes.

I use identical code in my driver to send custom CDBs. I can see using IORegistryExplorer that in USB 2 mode, macOS chooses alternate 0, the bulk-only protocol. My custom CDBs and their accompanying data pay loads are put on the bus, more or less as expected.

In USB 3 mode, macOS chooses alternate 1, the UAS protocol. My custom CDB is put on the bus, but no payload data is transferred.

Is this expected behavior? If so, is there a way to force the OS to choose alternate 0 even when on USB 3, perhaps with another dext?

I'll file a bug about this when Feedback Assistant lets me.

Answered by DTS Engineer in 860954022

Jumping around a bit, partly from the bug, partly from your post. First off, on your provider:

For USB2, this is IOSCSILogicalUnitNub, for USB 3 it is IOSCSIHierarchicalLogicalUnit.

Basically, you're trying too hard. Both drivers are subclasses of IOSCSIPeripheralDeviceNub, which is what you should use as your provider.

In USB 3 mode, macOS chooses alternate 1, the UAS protocol. My custom CDB is put on the bus, but no payload data is transferred. Is this expected behavior?

No. This should either work or we shouldn't let your driver load. I'm not sure WHY it's not working, but I don't see any obvious failure.

If so, is there a way to force the OS to choose alternate 0 even when on USB 3, perhaps with another dext?

I haven't tried it, but yes, I think this is possible. For the future, the way to figure this out is to start by tracking down its match dictionary in the underlying KEXT. Its bundle id (from the registry snapshot) is "com.apple.iokit.IOUSBMassStorageDriver", which is "IOUSBMassStorageDriver.kext". Here is the UAS entry from that dictionary:

<key>IOUSBMassStorageUASDriver</key>
<dict>
	<key>CFBundleIdentifier</key> = <string>com.apple.iokit.IOUSBMassStorageDriver</string>
	<key>IOClass</key> = <string>IOUSBMassStorageUASDriver</string>
	<key>IOProviderClass</key> = <string>IOUSBMassStorageDriverNub</string>
	<key>Read Time Out Duration</key> = <integer>30000</integer>
	<key>Retry Count</key> = <integer>20</integer>
	<key>Write Time Out Duration</key> = <integer>30000</integer>
	<key>bInterfaceProtocol</key> = <integer>98</integer>
</dict>

If you compare it to the non-UAS case, you'll find that "bInterfaceProtocol" is the key value. I haven't actually tried this, but I believe a codeless DEXT on IOUSBHostInterface that flipped bInterfaceProtocol from 0x62 (UAS) to 0x50 (from your working case) would do what you want.

  1. Every time I re-attach the device, I get a new driver process and a new IOUserServer at the root of the IORegistry. Even if I delete the app, the existing driver process keeps running, unless I kill it from Activity Monitor. Maybe there's something I'm not cleaning up properly, but I don't know what.

I think this is a bug in your code. Similar to KEXT, DEXTs aren't "unloaded" (meaning, the system kills them), they unload "themselves". I'm not sure how "done" your implementation is, but my usual approach with these is to pull out code until it unloads cleanly, then add code back in a piece at a time until I've reconstructed the "full" driver. You can also try logging out "all" of the lifecycle methods, but my experience has been that this will often tell you what the problem is ("stop isn't being called") without telling you WHY the problem is happening ("what you actually need to change"). Finally, you can also try running "lsmp" (command line tool) against one of those orphaned processes. That will give you a list of all its mach ports, which might be helpful if something "odd" is going on.

  1. When I eject the disk, there is a very perceptible delay between ejecting the volume and its associated - Data container. I use Finder/Settings and check the External Disks item in the Sidebar/Locations list. After I click the eject control next to the volume, I unplug the device as soon as it disappears from the sidebar, and very often see a notification about ejecting "volume - data" when it wasn't ready to be ejected.

If I had to guess, this is actually a symptom of #1, though I'm not sure exactly what. Is your DEXT on the I/O path? Does unmount trigger any DEXT activity?

Is this a bug worth filing?

So, arguably, that fact that this is happening is a bug:

After I click the eject control next to the volume, I unplug the device as soon as it disappears from the sidebar, and very often see a notification about ejecting "volume - data" when it wasn't ready to be ejected.

...since the icon shouldn't disappear until the kernel is "done". However, the problem is that the main thing that causes this is storage driver issues* or bad hardware, which is why you don't see it very much.

*I say this from personal experience (many years ago), having specifically had a storage driver that was "doing this" for a while it was in development.

BTW, if you're curious about what's triggering this dialog, the code is here.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Filed FB20382589

I'm not sure what's going on here.

Can you upload IORegistryExplorer files for the two cases (working and failure) to your bug? Saved from the app please, not the ioreg tool. Working with ioreg text files is always a pain.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

I uploaded the requested IORegistryExplorer files.

I also note a couple of other things:

  1. every time I re-attach the device, I get a new driver process and a new IOUserServer at the root of the IORegistry. Even if I delete the app the existing driver process keeps running, unless I kill it from Activity Monitor. Maybe there's something I'm not cleaning up properly, but I don't know what.

  2. when I eject the disk, there is a very perceptible delay between ejecting the volume and its associated - Data container. I use Finder/Settings and check the External Disks item in the Sidebar/Locations list. After I click the eject control next to the volume, I unplug the device as soon as it disappears from the sidebar, and very often see a notification about ejecting "volume - data" when it wasn't ready to be ejected.

If I keep /Volumes open I can see that "Volume"'s icon disappears first, then "Volume - Data"'s icon.

Is this a bug worth filing?

Jumping around a bit, partly from the bug, partly from your post. First off, on your provider:

For USB2, this is IOSCSILogicalUnitNub, for USB 3 it is IOSCSIHierarchicalLogicalUnit.

Basically, you're trying too hard. Both drivers are subclasses of IOSCSIPeripheralDeviceNub, which is what you should use as your provider.

In USB 3 mode, macOS chooses alternate 1, the UAS protocol. My custom CDB is put on the bus, but no payload data is transferred. Is this expected behavior?

No. This should either work or we shouldn't let your driver load. I'm not sure WHY it's not working, but I don't see any obvious failure.

If so, is there a way to force the OS to choose alternate 0 even when on USB 3, perhaps with another dext?

I haven't tried it, but yes, I think this is possible. For the future, the way to figure this out is to start by tracking down its match dictionary in the underlying KEXT. Its bundle id (from the registry snapshot) is "com.apple.iokit.IOUSBMassStorageDriver", which is "IOUSBMassStorageDriver.kext". Here is the UAS entry from that dictionary:

<key>IOUSBMassStorageUASDriver</key>
<dict>
	<key>CFBundleIdentifier</key> = <string>com.apple.iokit.IOUSBMassStorageDriver</string>
	<key>IOClass</key> = <string>IOUSBMassStorageUASDriver</string>
	<key>IOProviderClass</key> = <string>IOUSBMassStorageDriverNub</string>
	<key>Read Time Out Duration</key> = <integer>30000</integer>
	<key>Retry Count</key> = <integer>20</integer>
	<key>Write Time Out Duration</key> = <integer>30000</integer>
	<key>bInterfaceProtocol</key> = <integer>98</integer>
</dict>

If you compare it to the non-UAS case, you'll find that "bInterfaceProtocol" is the key value. I haven't actually tried this, but I believe a codeless DEXT on IOUSBHostInterface that flipped bInterfaceProtocol from 0x62 (UAS) to 0x50 (from your working case) would do what you want.

  1. Every time I re-attach the device, I get a new driver process and a new IOUserServer at the root of the IORegistry. Even if I delete the app, the existing driver process keeps running, unless I kill it from Activity Monitor. Maybe there's something I'm not cleaning up properly, but I don't know what.

I think this is a bug in your code. Similar to KEXT, DEXTs aren't "unloaded" (meaning, the system kills them), they unload "themselves". I'm not sure how "done" your implementation is, but my usual approach with these is to pull out code until it unloads cleanly, then add code back in a piece at a time until I've reconstructed the "full" driver. You can also try logging out "all" of the lifecycle methods, but my experience has been that this will often tell you what the problem is ("stop isn't being called") without telling you WHY the problem is happening ("what you actually need to change"). Finally, you can also try running "lsmp" (command line tool) against one of those orphaned processes. That will give you a list of all its mach ports, which might be helpful if something "odd" is going on.

  1. When I eject the disk, there is a very perceptible delay between ejecting the volume and its associated - Data container. I use Finder/Settings and check the External Disks item in the Sidebar/Locations list. After I click the eject control next to the volume, I unplug the device as soon as it disappears from the sidebar, and very often see a notification about ejecting "volume - data" when it wasn't ready to be ejected.

If I had to guess, this is actually a symptom of #1, though I'm not sure exactly what. Is your DEXT on the I/O path? Does unmount trigger any DEXT activity?

Is this a bug worth filing?

So, arguably, that fact that this is happening is a bug:

After I click the eject control next to the volume, I unplug the device as soon as it disappears from the sidebar, and very often see a notification about ejecting "volume - data" when it wasn't ready to be ejected.

...since the icon shouldn't disappear until the kernel is "done". However, the problem is that the main thing that causes this is storage driver issues* or bad hardware, which is why you don't see it very much.

*I say this from personal experience (many years ago), having specifically had a storage driver that was "doing this" for a while it was in development.

BTW, if you're curious about what's triggering this dialog, the code is here.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Should UserSendCBD work on UAS interfaces?
 
 
Q