BlockStorageDeviceDriverKit grant confirmed by support but shows "No Requests" in the portal. How to resolve?

Hello!

I am hoping a DTS engineer or someone who knows the Capability Requests portal can help, because I am stuck between a written support confirmation and what the portal actually shows.

Background. We are building a native macOS iSCSI initiator for SOHO and home NAS use, developed over close to two years. A userspace daemon runs the iSCSI protocol and a DriverKit system extension presents the remote LUN as a block device. The code is essentially complete. Only the DriverKit extension cannot be signed, loaded and validated without the entitlement.

We submitted request 32PC8MGU57 for two entitlements: com.apple.developer.driverkit.family.block-storage-device for the extension com.aviontex.iscsi.AviontexISCSI.AviontexInitiator com.apple.developer.driverkit.userclient-access for the app com.aviontex.iscsi.AviontexISCSI, scoped to the extension bundle id

The problem. On June 25 Developer Support confirmed in writing that both entitlements were granted. The portal does not match that: Block Storage Device: No Requests: on both App IDs UserClient Access: Assigned: on the app SCSI Controller: Submitted: on the app

So the one entitlement we actually need, Block Storage Device, shows as never requested, even though request 32PC8MGU57 covered it and support confirmed the grant. The case was escalated to the senior team on July 2 (case 102922935570). Follow-up emails since then have not received a response.

Why Block Storage Device specifically

Our initiator has no PCI or Thunderbolt bus and no DMA path, so SCSIControllerDriverKit does not fit. This is confirmed by DTS in thread 776020, where Kevin Elliott explains that SCSIControllerDriverKit passes data through fBufferIOVMAddr as a physical address with no mechanism to convert it into a VM address the dext can access. He also notes it cannot be used with any bus other than PCI or Thunderbolt. Block Storage Device is therefore the family we need.

My questions: Am I reading the portal correctly: Block Storage Device not requested, UserClient Access assigned, SCSI Controller submitted? From here, what is the correct way to get Block Storage Device onto these two App IDs, with both the Development and the Distribution grant, since our public beta depends on Distribution? Should I submit a new request through the Capability Requests tab or does the escalated case handle it? Is there any way to get visibility on the escalated case, since email follow-ups are not being answered?

A full technical justification is prepared and we are happy to share the source code. Any guidance would be appreciated.

Thank you.

Answered by DTS Engineer in 915504022

I'm extremely limited in what I can share, but what I'll say is the following:

I can now tell you that our support for software I/O has been released in first seed of macOS 27.3. I have an extended post about it here, but the basic summary is that all you need to do is add an additional property when you call UserSetHBAProperties before controller initialization and then it will "just work".

Please take a look at test with this on Apple Silicon. I'm confident in the basic architecture and we have done some testing on our side, but if there are any problems they need to be identified and reported as quickly as possible so I can try to get them fixed before 27.3's final release.

Give it a try and let me know what you find!

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Hi Kevin,

thanks a lot for the quick view and answer. You went through all the Feedback reports remarkably quickly. Huge respect for that.

Q1. Why did you choose a 4KiB maximum and 16KiB alignment for FB25095303? What were you trying to establish?

We first tried the 16KiB maximum, 16KiB alignment and one-segment configuration you had suggested. That led us to the 4KiB test.

When nothing makes sense, I sometimes like to try something seemingly pointless to confirm the pointlessness. So we stepped back into abstraction and asked where the “famous” single difference between Intel and Apple Silicon was. We focused on 4KiB versus 16KiB and tried the Intel settings on Apple Silicon. The M5 kernel’s 16KiB page size was the one thing we could not change. It was a direct comparison.

In the 4KiB diagnostic build, the host-facing maximum read and write byte counts and maximum segment byte counts were also 4KiB. Its captured IORegistry reports two HBA segments.

Q2. What happened in the 16KiB/16KiB test? Which filesystem did you format or mount and what exactly failed?

Only RAW access worked. Formatting and mounting preformatted LUNs began but then nothing happened. There was no completed filesystem mount, no error and no panic. We did not investigate where the operations stopped.

In separate tests, even exFAT caused a panic. That led us to add strict read-only mounting to the initiator. With that enabled, all the filesystems we tried mounted. Writes and formatting were rejected, including format commands issued from the command line.

Q3. How did you measure approximately 0.08 Gbit/s?

We used RAW writes, reads and byte by byte verification. The data matched and we saw no EIO or panic during the 16KiB/16KiB test. We were primarily checking for those failures. At roughly 0.08 Gbit/s, we quickly dropped that configuration.

I expected your answer about the KEXT entitlement and was pretty sure from the start. However, Micha from my team thought I should ask you anyway. Well, I won that bet. ;)

The KEXT was an experiment to answer one question: can we avoid the panic at all? With it in place, we could read, write and verify without errors or panics. Even hours-long write or read tests completed without a single EIO.

We have not yet run a dedicated test under heavy memory pressure to stress DMA and memory. At this stage, our main question was whether the approach worked at all. The results show us that we are moving in the right direction, but I agree with you that this is a useful proof rather than a solution. From my point of view, needing a KEXT just to get a DEXT working is almost an oxymoron.

That is also why we included the KEXT source with the Feedback reports. It is only meant to show how we worked around the panic in a “quick & dirty” way. Perhaps it will be useful to your team as well.

Best greetings!!

Torsten

Hi Kevin,

just for clarification:

I'm sorry, but that isn't going to happen. We're very unlikely to grant them at all today, and we've never granted them for testing purposes.

Does this mean we can get the KEXT entitlement for a rollout until everything is done on your end?

If YES and that were possible, that would be awesome.

Naturally, we wouldn't just use it for testing then and we would only need it until everything is fixed.

Best greetings,

Torsten

So, in the process of writing up a response, I ended up getting pulled into a bit of a side chain which ended up invalidating everything I'd written.

The bottom line is that I think you (and I) have been looking at this a bit wrong. The basic assumption here has always been that single-page I/O was inherently a serious performance issue (which is still probably somewhat true) that doesn't "work,” but the problem is that it turns out that one of our drivers is actually doing I/O with an individual transfer size of 4k and an overall configuration VERY similar to what yours should be.

Here are the key parameters I just found looking at the NVME controller of the MacBook Pro I'm currently writing this on:

kIOMaximumSegmentByteCount = 0x1000 -> 4096 bytes
kIOMaximumSegmentCount = 0x100 -> 256 segments
kIOMaximumSegmentByteCount * kIOMaximumSegmentCount = 1,048,576 bytes

IOMaximumByteCount = 0x100000 = 1,048,576 bytes

Now, the in-kernel NVMe controller is obviously very different than your situation, but it's not THAT different. I don't know if the performance will be good enough to ship, but there's certainly no reason why it shouldn't at least "work.”

With that context, looking at the sysdiagnoses you sent with your bug reports, I think there are serious issues with your registry configuration. You've got this set:

"IOMaximumByteCountRead" = 2080768
"IOMaximumSegmentByteCountWrite" = 2080768

...and I think that's just "wrong.” I don't think you should/need to set IOMaximumByteCountRead at all, but I think your basic parameters should be:

kIOMaximumSegmentByteCount = 0x4000 -> 16,384 bytes (1 page)
kIOMaximumSegmentCount = 128 (your current size, but it could be larger)

Which would then make:

IOMaximumByteCount = 0x4000 * 128 = 2,097,152 = 2048k

Note that I'd still recommend NOT setting IOMaximumByteCount. It's not necessary and really only has one valid value (the calculation above), which the system will take care of for you.

A few other things to look at here:

  • I haven't seen a registry log that includes the entire IOMedia stack, but I would strongly recommend that you go through that stack, looking at all the keys that control I/O request management and verify that they "make sense.” I don't think they do, and I think that's where things are breaking down.

  • On the testing/validation side, I think I would try opening the buffered device node ("disk,” not "rdisk") and trying to do "randomly" sized I/O "all over" the disk. IF your device configuration is set up properly, that should work fine, as the system will generate block-aligned requests and then return the data segment you asked for. If it doesn't work, then that indicates that your device configuration isn't right. FYI, I think this could be a problem:

"IOMinimumSegmentAlignmentByteCount" = 4

  • One other thing to try here would be to increase the logical block size you present to 16k. I don't think that's actually required, but it is the most direct way to eliminate all unaligned I/O.

Related to all this:

In separate tests, even exFAT caused a panic.

and

Only RAW access worked.

What did the panic actually show? One thing to understand here is that, as far as IOKit is concerned, and your driver in particular, there isn’t a difference between the raw I/O and the buffered I/O the system performs. It's all just blocks that you're shuffling about. That means any difference you're seeing is caused by the VFS layer sending different requests than "you" did.

Note that this:

Formatting and mounting preformatted LUNs began but then nothing happened.

...is what we've seen happen before when a request reaches IOUserSCSIController that it's unable to generate a segment pointer for. The whole "point" of your driver's configuration should be ensuring that does not happen.

That leads to here:

We used RAW writes, reads, and byte-by-byte verification.

Were those single-page, 16k writes? Or did you also test large writes? I think what might be going on here is that your device configuration isn't actually constraining your I/O like you need it to, but your RAW tests only sent requests that fell within your constraints. You'll then blow up as soon as your requests move outside of those constraints.

Similarly, my guess is that if you run this same driver on Intel, you'll find that you’re getting larger requests than you "should.” That's exactly the same problem; it's just not failing the way it would on Apple Silicon.

Does this mean we can get the KEXT entitlement for a rollout until everything is done on your end?

No.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Hi Kevin,

This is a little difficult at the moment because we cannot take any measurements on the Apple Silicon machine until the end of October. My colleague is currently on vacation in Brazil, visiting his in-laws. However, I’ll try to reconstruct what happened from memory.

We used exactly the values you suggested. This resulted in extremely slow performance, and only RAW access worked. Mounting the filesystem was not possible. Yes, we tried the 16 KiB configuration, but that did not get us any further. If it helps, I can send you a complete log showing exactly what happens, where, how, and when, but that will take a few days.

There is also the maximum total read/write transfer size to consider. Explaining this in detail here in the forum would mean giving away a technical advantage, so if anything is unclear, please ask me to clarify.

When the data passes through the kernel, Apple adds a header. It is small, but accounting for it is extremely important. Interestingly, it differs between Intel and Apple Silicon. So yes, we can use the system values, but doing so produces large numbers of errors or, depending on the configuration, read/write or completion errors. The kernel also frequently logs EIO errors. There is no panic, but we then see SCSI errors—sometimes just a few, sometimes a large number. There is no additional logging of this except at one specific point when an error occurs.

That is why simply choosing 1 MiB or 2 MiB does not work reliably. Some additional calculations are necessary to make this work correctly and consistently.

Does this help for now?

Best greetings,

Torsten

Hi Kevin,

we have to accept the “no” regarding KEXTs, but at the moment there is no viable path forward for iSCSI on Apple Silicon. After a brief internal discussion, we have decided to halt development and take it off the table until this works without KEXT.

Personally, I don’t see a way forward for at least the next six to nine months, so further development makes no sense at this point. Even the iSCSI mathematics involved are completely different with and without a KEXT.

Nevertheless, thank you very much for your incredible support!

Best greetings!!

Torsten

We used exactly the values you suggested. This resulted in extremely slow performance, and only RAW access worked. Mounting the filesystem was not possible. Yes, we tried the 16 KiB configuration, but that did not get us any further.

OK, but understand that the only values I've actually seen were simply wrong on both Intel and Apple Silicon. They wouldn't have directly failed on Intel but would mean that the system was not working as intended.

When the data passes through the kernel, Apple adds a header. It is small, but accounting for it is extremely important.

I'm not sure what you're talking about here. If you're comparing direct I/O (through the rdisk node) to file system I/O, then you're right that there isn't a one-to-one correspondence between the direct reads a user initiates and what the file system actually sends to disk. That's partly tied to the file system’s own structure and partly caused by how read caching works in the UBC. However, that's not really a "header“; it's just doing extra I/O.

Interestingly, it differs between Intel and Apple Silicon.

I've never looked at it in detail, but, yes, I'd expect the larger page size to alter the details of exactly what I/O the UBC issues.

So yes, we can use the system values, but doing so produces large numbers of errors or, depending on the configuration, read/write or completion errors. The kernel also frequently logs EIO errors.

I believe EIO is what would result from us failing to generate fBufferIOVMAddr, however, your I/O configuration should be structured such that you never receive an I/O larger than 16k, so wiring can't fail. One easy test here— have you tried validating your Apple Silicon configuration on Intel? You can then run every possible test, and if you ever receive a request larger than 16k, you'll then know you've got a problem.

We have to accept the “no” regarding KEXTs, but at the moment there is no viable path forward for iSCSI on Apple Silicon.

Yes, I agree with that.

Personally, I don’t see a way forward for at least the next six to nine months, so further development makes no sense at this point.

Please keep an eye on this thread, as I'll update it as soon as anything changes.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Hi Kevin,

thank you for the detailed reply. I think we need to separate Apple Silicon from Intel.

On Apple Silicon, I agree with you. We tried many configurations while trying to make filesystem mounting work with the DEXT alone. Those settings were largely experimental. As long as a filesystem mount or subsequent I/O can trigger a kernel panic, further iSCSI tuning makes little sense. A reliable filesystem mount has to come first.

On Intel, I disagree that the current implementation is not working as intended. Our initiator mounts filesystems and has passed byte-exact data integrity and memory-pressure tests without EIO or kernel panics. The current benchmark reaches about 16 Gbit/s. We could pursue more throughput, but the extra CPU load would hurt other work on the Mac. For Intel NAS use, this is a balance we’re happy with.

When I used the word “header,” I meant the extra room we observed at the Intel transfer boundary inside the kernel, not additional filesystem I/O generated by the UBC. We did not measure or trace precisely where that margin comes from, since we cannot change that kernel behavior. Possible contributors include page alignment and padding, I/O mapping constraints, network hardware alignment, or bookkeeping associated with mbufs and memory descriptors. Ignoring the margin can cause errors or a sharp drop in throughput, so our Intel configuration accounts for it. In summary: “We don’t know what it is, but we accept it.” ;)

Anyway, here’s my proposed deal:

If you and your team can help make filesystem mounting and subsequent I/O reliable through the DEXT alone on Apple Silicon, without kernel panics, we’ll take care of getting the iSCSI side right. My personal goal is 105 Gbit/s. Our quick-and-dirty KEXT-assisted prototype already reached 81 Gbit/s in the RAW benchmark, so I’m curious to see what a reliable DEXT-only implementation can achieve. Then we can happily argue about geometry again. ;)

Please keep us posted. As long as a matching KDK is available for the macOS build, we’d be happy to help you and your team at any time.

Best greetings,

Torsten

Hi Kevin,

I have a small correction for you because it was bothering me and would not let me sleep. I have no idea what went wrong there.

You were absolutely right. If we align it 100% according to your specifications, a mount is indeed possible, regardless of the filesystem. However, as you also mentioned, we are light years away from iSCSI performance with this.

Read speeds are around 30 MiB/s and write speeds are around 26 MiB/s.

So these iSCSI speeds are roughly 20x faster than USB 1.1, at least that is something, but still slower than USB 2.0. That is definitely not even a start ;)

Best regards,

Torsten

You were absolutely right. If we align it 100% according to your specifications, a mount is indeed possible, regardless of the filesystem. However, as you also mentioned, we are light years away from iSCSI performance with this.

OK, good.

I think I made this clear earlier, but my intention here has been to get you to a reliable, working implementation, NOT one that was fast enough to actually be useful. That working implementation lets you validate and test as much as you possibly can "now,” which should allow you to update and very quickly ship a working solution as soon as we provide a software I/O flow. I'm sorry that I can't be more specific than that, and I appreciate your understanding and patience.

I didn't expect a fast implementation to be possible, and the fact that the Intel approach works the way it does was a surprise to me.

So these iSCSI speeds are roughly 20x faster than USB 1.1, at least that is something, but still slower than USB 2.0.

Ironically, I'm old enough to remember when this:

Read speeds are around 30 MiB/s and write speeds are around 26 MiB/s.

...was very respectable SCSI performance, back in the good old days when any "real" Mac had a stack of SCSI drives piled next to it.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Hi Kevin,

Thanks, that clears things up and makes perfect sense.

To be fair, our Intel implementation still has rough edges too. My advantage there is that I've known Intel since 1993 and the kernel handles that platform exactly the way it should. That leaves several paths open. Whether Apple Silicon offers the same flexibility, I can't say yet: I simply don't know that CPU well enough.

Of course it bugs me that arm64 isn't there yet. On the upside, it lets us start with Intel users first. That's a small crowd, but big enough to polish the GUI and the core iSCSI functions. Odd as it may sound, Apple Silicon is a completely different beast from Intel and iSCSI makes that difference very visible. So we get to take one step at a time.

A reliable baseline is exactly what we need right now. We'll use it on Apple Silicon for full validation: byte-exact data integrity, memory pressure, sleep/wake and network reconnect cycles plus real backup workloads. Once the software I/O flow arrives, we can focus purely on throughput.

Having done some kernel development myself, I know these are anything but quick fixes you push out and then wish everyone good luck. This is the heart, brain and central nervous system of macOS all at once. So no worries, I fully understand that it takes time.

And yes, I'm old enough to remember those days too. A stack of SCSI drives next to the Mac, terminators and juggling SCSI IDs. Back then 30 MiB/s would have been a dream, so in a way iSCSI on Apple Silicon has come full circle. Maybe we should release an iSCSI4NAS Retro Edition: Apple Silicon only, with a 30 MiB/s retro speed limit included. ;)

Anyway, thanks again for your patience and support. We'll keep an eye on this thread and our offer stands: as long as a matching KDK is available, we're happy to help with testing any time.

Best regards,

Torsten

Hi Kevin,

here’s a quick update: since you already mentioned that the fixes will take a while longer, we had a few more discussions with the Mac, the kernel and the CPU ;)

If we can get this running stable above 2.5 Gbit/s, we’ll at least have a foundation for an iSCSI initiator on Apple Silicon. You can’t exactly call that “performance”, but for most home users 1 or 2.5 Gbit is the ceiling anyway.

Better than nothing or waiting forever. And definitely better than FSKit or SMB crap.

Hi Kevin,

We prepared an iSCSI build for TestFlight. The app uses a separate native Core process for security reasons and the SCSI extension. The Core maintains active storage connections independently of the GUI.
Compilation, distribution signing, packaging and notarization all succeeded.

Now the problem: App Store Connect rejects the upload with two errors:

Error 90296: App Sandbox is required for the GUI, Core and helper executables.

Error 90941: Missing DriverKit entitlement, although com.apple.developer.driverkit is present.

  1. Moving this entire architecture into App Sandbox while preserving its required functionality is technically impossible. We are talking about iSCSI, mounting and formatting LUNs. If this were possible using App Sandbox, it would prove that Apple has serious problems. It’s logically and technically impossible.

  2. This is definitely a false positive because the entitlement is present. The app has also been successfully notarized and installs and runs correctly on another Mac with a clean installation.

Does this mean our iSCSI initiator cannot be distributed through the Mac App Store, even though we otherwise fully comply with all of Apple's requirements for this iSCSI initiator?

Is there a way to get some kind of exception?

Best regards,

Torsten

Moving this entire architecture into App Sandbox while preserving its required functionality is technically impossible. We are talking about iSCSI, mounting and formatting LUNs. If this were possible using App Sandbox, it would prove that Apple has serious problems. It’s logically and technically impossible.

Nah, that's not true. The problem here is that the term "App Sandbox" is very commonly used to refer two VERY different things:

  1. The system that allows apps to pre-declare what their app "does", which the system will then enforce.

  2. The particular set of limitation/rules the Mac App Store requires apps to adhere to.

The critical point here is that #1 doesn't necessarily prevent an app from doing ANYTHING, as the role of #1 isn't to allow apps to declare "what they're going to do", not constrain them from actually doing it.

Have you tried turning on the app sandbox and, if so, what problems are you actually having?

This is definitely a false positive because the entitlement is present. The app has also been successfully notarized and installs and runs correctly on another Mac with a clean installation.

First off, take the time to manually export the build and make SURE its including what it should, both in terms of the specific entitlement and other entitlements. Different build configuration use different signing setups, which can end up changing final signing configuration. Similarly, make sure you check "everything", not just the specific issue Xcode is pointing at. In my experience, Xcode is pretty good at recognizing invalid configuration and VERY bad at describing whats actually wrong with the configuration. Start with the assumption that there is a problem and its not what Xcode told you.

Does this mean our iSCSI initiator cannot be distributed through the Mac App Store, even though we otherwise fully comply with all of Apple's requirements for this iSCSI initiator?

I'm not sure. The biggest issue here is that the Mac App Store has a blanket rule against app presenting authorization dialogs and/or executing code a root, but I don't know enough about the specific of your app to know how that effects you.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Hi Kevin,

I need to revise my earlier assessment completely. I seriously underestimated what App Sandbox can do. My teammate Micha had already proven it. I am now officially outnumbered ;) I guess I'm getting too old for this shit.

The only thing that doesn't work yet is formatting LUNs from within the app. It would be great to have this supported as well, just like with USB devices. For now we can live with it. We simply haven't found a suitable API for doing this inside the sandbox. If there is one, please let me know.

Honestly, "Sandbox" is a serious understatement for what you built there. "SandOS" would fit much better ;)

One example: we have now decoupled our processes with the sandbox. GUI and DEXT run separately.

  • The GUI crashes? iSCSI keeps running.
  • The DEXT crashes? It is back so fast that iSCSI doesn't even notice. (!)
  • Both crash? The DEXT recovers. iSCSI holds. The GUI stays closed. Once it is reopened, everything looks as if nothing had happened.

Whatever happens in our stack, the storage connection stays intact as long as the OS is running. That is really incredible work by your teams. It opens up completely different possibilities for us.

Thank you for challenging my assumption.

Best regards,

Torsten

The only thing that doesn't work yet is formatting LUNs from within the app. It would be great to have this supported as well, just like with USB devices. For now we can live with it. We simply haven't found a suitable API for doing this inside the sandbox. If there is one, please let me know.

So, what are you actually "doing" here? Is this about your target side app trying to write to physical media? Or is this your initiator trying to write a new file system to a disk you're presenting?

If you're trying to write to dev nodes, try giving your app com.apple.security.exception.files.absolute-path.read-write to "/dev/". I don't know if that will be sufficient to give you full read/write, but it's an easy thing to check.

Honestly, "Sandbox" is a serious understatement for what you built there. "SandOS" would fit much better ;)

Yeah, it's pretty comprehensive. One of those accidents of history is that because the public "introduction" of the sandbox happened at the same time as the Mac App Store, the focus of our documentation was on how apps should alter their implementation to better conform to those expectations.

However, what we never really wrote/created was a truly comprehensive set of API/docs that described how anything could be described/constrained. That's unfortunate as it really is the case that every app should be sandboxed, even processes that do have substantial security implications. For example, an app may need/want to access every file on your file system but that doesn't mean it needs any access to IOKit. There's no reason the app shouldn't be able to tell us that.

One example: we have now decoupled our processes with the sandbox. GUI and DEXT run separately.
The GUI crashes? iSCSI keeps running.
The DEXT crashes? It is back so fast that iSCSI doesn't even notice. (!)

Yeah, it's easy to underestimate just how fast IOKit operates in the kernel. The entire IOService tree ends up living in wired memory and basically all of it has been subject to 26+ years of optimization. Honestly, the slowest part of DEXT initialization is launching your process and even that has been heavily optimized as the system collection of chattering daemons has grown and grown.

Both crash? The DEXT recovers. iSCSI holds. The GUI stays closed. Once it is reopened, everything looks as if nothing had happened.

Well done. That's the "ideal" for this sort of thing, but pulling it off can definitely be tricky.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Hi Kevin,

thank you for the kind words. A “well done” from you means a lot.

About formatting, I phrased that badly. I meant plain macOS formatting: the user erases the LUN the standard way, exactly as with a USB drive. That is what we would like to offer from inside the app.

Your /dev/ suggestion is on my list and I will let you know how it goes once I have tested it.

I also have a confession. Over the weekend I cursed Apple. After I installed Beta 3, our iSCSI initiator on Apple Silicon didn’t just die, it took everything down with it. Mount, format and read all failed. I still have to take a closer look, but if this is what I think it is, that would be huge. In that case breaking everything was totally worth it ;) We will test all of it as fast as we can.

Honestly, I expected this to take a year and a half to two years. Getting it this soon is more than just fantastic. A thousand thanks and keep it up!!!

Best regards,

Torsten

I'm extremely limited in what I can share, but what I'll say is the following:

I can now tell you that our support for software I/O has been released in first seed of macOS 27.3. I have an extended post about it here, but the basic summary is that all you need to do is add an additional property when you call UserSetHBAProperties before controller initialization and then it will "just work".

Please take a look at test with this on Apple Silicon. I'm confident in the basic architecture and we have done some testing on our side, but if there are any problems they need to be identified and reported as quickly as possible so I can try to get them fixed before 27.3's final release.

Give it a try and let me know what you find!

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Hi Kevin,

Just wanted to say a massive thank you to you and the team for macOS 27.3 and the DriverKit update. Seeing FB24326124 right there in the release notes made our last two days.

In short words: YESSSSSSS!!!! IT’S WORKING!!!!


(M5, DEXT iSCSI - and no optimization)

Thanks for the clarified ownership contract in FB24326124 and the new software-controller support. So, on the software-I/O path, we've got the tag-check panics from FB25095318 and FB25095303 sorted, and we're also good to go with the unaligned-buffer issue from FB25095286.

Mounting and formatting are now reliable with APFS and exFAT. Everything's running smoothly on our end.

And what about that offer you made on Tuesday? That was a massive deal. We got down to business right away, and well, let's just say we didn't hold back. It's been three nights since then and sleep's been pretty much out of the question ;) It's definitely worth it.

So let's do this. We'd love to take a look at these. I've put them into three groups: urgent (you've really got to take a look at these), helpful (these will improve performance) and our wishlist. It'd be great if you could fix these.


You should definitely check this out:

FB25545328
FB25545357
FB25545390
FB25545420

This would be helpful for us:

FB24597050 (updated)

And since Christmas is just around the corner, here's our little wish list of things we'd like:

FB25545444

The technical details are in the individual reports. Let me know if you need more or detailed informations or reports.

Anyway, a big thanks again for listening and for pushing this forward to the Team!!!!!!

Best greetings,

Torsten

So, working through the bug list, I'm going to start here:

FB25545390

This question was about serialization callback order at teardown time. The thing to understand here is that the nature of the SCSI controller architecture means that teardown is always initiated by the hardware target being destroyed, never by user space. That is, in a "clean" teardown on physical hardware, what actually happens is:

  1. User space unmounts all volumes.
  2. All I/O buffers are flushed to disk.
  3. All I/O completes. No new I/O commands occur because nothing is using the device.
  4. The user unplugs the device.
  5. The SCSI controller tears everything down.
  6. The SCSI controller destroys itself.

...and a "dirty" teardown just means we started at #4 instead of #1. I outline all this because if you're implementing "intentional" teardown, you may want to consider initiating the "clean" flow from user space instead of just relying entirely on the hot plug.

Now, the most critical thing to understand here is that everything that happens from #5 on is under the controller's control, not something that was "forced" onto it. Indeed, nothing actually prevents the SCSI controller from never tearing down AT ALL. That's even more true in your case—your provider is never going to go away at all, so you have complete control over how the entire process plays out.

Now, that leads to the most important detail here: DriverKit::Stop() is NOT equivalent to IOKit::stop(). IOKit::stop() occurs close to the end of the teardown process and basically marks the final end of "normal" operation. It's also a synchronous call, so returning from "stop" is a hard endpoint.

DriverKit::Stop() is basically the complete opposite of that, as it occurs at the very BEGINNING of teardown and marks the point where your DEXT is being told that Stop is occurring. At a purely technical level, it's actually called by IOService::willTerminate, which happens at the very beginning of termination. Most importantly, it's an asynchronous call, NOT a synchronous one. The documentation is actually quite clear on this point and how it relates to I/O:

"If your driver has any in-progress asynchronous tasks, cancel those tasks and wait for DriverKit to call the associated cancellation handler before calling the super version of this method."

In other words, don't call "super::Stop" until you're "totally done".

There's a similar detail here:

"Call super at the end of your implementation."

This is often read as "at the end of self::Stop()", but that's NOT what it said or meant. What it actually meant was, "return from self::Stop()" and call "super::Stop()" whenever you're actually done.

Finally, there's an implication to this detail that's often overlooked:

"After you call super, it is a programmer error to access the provider object."

The inverse is that you're allowed to call your provider up until the point you call super, which means until you call super::Stop, your DEXT is still operating relatively "normally".

In any case, the key point here is that all of your teardown work should be initiated by "self::Stop()" and allowed to complete before calling "super::Stop". The vast majority of DriverKit teardown problems are caused by failing to follow that guidance.

FB25545420

So, the first thing I'll say here is that there's a critical design choice you need to make between:

  1. Using a single, long-lived SCSI controller that creates/destroys individual devices.
  2. Using multiple SCSI controllers which are bound to individual devices and are destroyed when the device is.

Architecturally, the first approach is the easiest to implement, as your SCSI controller can match against IOUserResource and then create/destroy devices whenever its user client tells it to. I think my own leaning would be toward #2 and, most critically, have SCSI controller instances live in individual DEXT processes. The advantage of this approach is that process separation means you can worry less about perfect resource management, and a failure on one device can't take down all devices.

In terms of implementation, what the second approach means is that you end up having three different classes at work:

  1. Your initial class which loads against IOUserResource.
  2. An object created by #1 using "Create()", which acts as the nub for #3.
  3. The actual SCSI controller, matched and loaded by #2 calling RegisterService().

Both 1 & 2 can live in the same process and are both relatively simple implementations. #3 is the complex case, but it sidesteps most resource management issues by allowing itself to be destroyed whenever you're "done" with a given device.

FB25545328
FB25545444

I've suggested dropping the extra bzero, but I think larger changes to the I/O path are unlikely, at least during macOS 27. We intentionally avoided touching DriverKit's implementation due to risk concerns, and that issue would apply here as well. If we're going to modify the I/O path, our goal would be to entirely remove the call to UserCopyDataBuffer and streamline communication all the way to user space, not to optimize this implementation. However, that would need to wait for a major release.

FB25545357

Why are you using IODMACommand at all? If you've set up a shared ring buffer with user space, then you don't need to use DMA at all. Just copy the bytes from your UserCopyDataBuffer into the shared buffer. I think IODMACommand is just making everything slower, probably more than the specific issues you pointed out.

FB24597050

Could you clarify the supported Task Management path for DriverKit SCSI controllers and whether these entry points can be enabled to reach the DEXT callbacks?

Sure. Our SCSI stack doesn't really support Task Management and never really has, which is why these methods are unimplemented. That is, if DriverKit fully supported them today, nothing would actually change, as the kernel doesn't call these methods, nor is there a user space channel that you could use to call them. They should be removed from DriverKit at some point (r.185932486), but I'm not sure if/when that will actually happen. The work itself is straightforward, but it carries binary compatibility risk, which means it keeps getting put off in favor of higher-priority work[1].

[1] For example, adding support for software I/O.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Hi Kevin!

Just between us: the speed at which you sometimes reply is almost a little scary. You’re not psychic? Do you have to memorize the entire kernel source shortly after being hired at Apple? You know what the word “hobby” means without having to google it? It’s uncanny how you do this. ;-) Seriously: RESPECT!

But back to the topic.

That’s it! We’re now at around 85 Gbit/s (RAW loopback) with zero remaining issues. Nothing at all. So if it were up to us, make Beta 1 the release and ship it. This is hereby approved on our side. ;-)

We were finally able to solve the critical problems with your recommendations and tips. It was clearly written there and we just didn’t see it. But now it looks good. Sure, there’s always room for further optimization, but a million thanks to you and the team.

I wasn’t quite sure how to properly bring all of this into the forum, so feel free to close the tickets we’ve resolved here. You can also mark the remaining open ones as resolved.

Just as a quick side note: once the DEXT is running really solidly (which should be soon), we already have a Mac Studio-based Virtual Tape Library planned (in headless mode). Just a heads-up for you. ;-)

With that said, a thousand thanks to you and your team for everything!!!

Best regards,

Torsten

Just between us: the speed at which you sometimes reply is almost a little scary. You’re not psychic? Do you have to memorize the entire kernel source shortly after being hired at Apple? You know what the word “hobby” means without having to google it? It’s uncanny how you do this. ;-) Seriously: RESPECT!

LOL. Well, thank you. I certainly don't have the kernel memorized, but I have been working on this "stuff" for what now seems like an appallingly long time. With IOKit in particular, I actually attended one of the 3 day "Introduction to IOKit" sessions Apple held (in ~2001?!?) to teach existing Mac developers about IOKit. I still have a very clear memory of telling IOKit's architect that it was so much better for having used C++ over ObjectiveC[1].

That’s it! We’re now at around 85 Gbit/s (RAW loopback) with zero remaining issues. Nothing at all. So if it were up to us, make Beta 1 the release and ship it. This is hereby approved on our side. ;-)

We were finally able to solve the critical problems with your recommendations and tips. It was clearly written there and we just didn’t see it. But now it looks good. Sure, there’s always room for further optimization, but a million thanks to you and the team.

Fabulous! I've passed that along to the mass storage team and, please give them the credit here not me. I did come up with idea for the flow we're using and contributed a few suggestion, but they did all of the hard work that made it happen.

I wasn’t quite sure how to properly bring all of this into the forum, so feel free to close the tickets we’ve resolved here. You can also mark the remaining open ones as resolved.

There isn't really any formal process, as our tooling let track specific threads and see any that have new changes I haven't read. For our purposes, just post anything new that comes up as a new thread, as three pages ought to be enough for anyone.

[1] I was a young idiot and ObjectiveC IOKit would have been FABULOUS. In hindsight, I respect him greatly for not having laughed in my face.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Hi Kevin,

the IOKit story is fantastic. As someone who grew up with C/C++ and Assembler, I probably would have celebrated C++ back then too, simply because it would have felt like pure luxury.

Back in those days I was on the dark side of the force myself.
Let’s make the things that break the things they made.
I think I don’t need to say more than that.

Nothing worked without Assembler back then. StormC on the Amiga, that’s where it all started. More Guru Meditation errors than anything that actually ran. Oh man, those were the days and without all that know-how of the past years a lot of what we did here would not have been possible.

It was a lot of fun working with you, especially your last post. We had been at it for almost 4 days and nights straight and you pull the answer out of your back pocket. Don’t ask how stunned I was. ;) I expected a lot but not a few hours. Anyway, right now we are “Go!” for public testing.

Please pass our thanks on to the mass storage team. The credit gladly goes to them but the idea for the flow came from you and exactly that flow untied the knot for us. Without both we would not be at 85 Gbit/s today.

Oh and one small note. Right now the Initiator runs on Intel starting with macOS Ventura but only with a whole lot of bypasses and tricks. If you could bring the patch into the Intel kernel as well, that would be a huge deal. It would make Time Machine backups over iSCSI much easier and with today’s hardware prices that would be a real feature. It would also make a cost effective migration from Intel to Apple Silicon over iSCSI possible. This is not official Feedback and I know Intel is hardly a focus for you anymore but I think it’s worth considering.

Anything new will come as its own thread from now on. Three pages really ought to be enough for anyone. Worked for 640K too. ;-)

Thanks for the patience, the fun and above all the incredible speed on your side. It was a real pleasure for all of us and especially for me.

Best regards from Germany!!!

Torsten

Oh and one small note. Right now the Initiator runs on Intel starting with macOS Ventura but only with a whole lot of bypasses and tricks. If you could bring the patch into the Intel kernel as well, that would be a huge deal.

I don't know. Porting a fix like this back to macOS 26 would be quite unusual, but if you file a bug asking for it then I'll make sure the team sees it.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware