Is DEXT Driver supporting these Networking Features?

Tx/Rx Multiqueue,

Yes. A DEXT can have multiple IOUserNetworkRxSubmissionQueue/IOUserNetworkRxCompletionQueue and IOUserNetworkTxSubmissionQueue/IOUserNetworkTxCompletionQueue.

RSS

The queue architecture above handles this. Every queue has an associated dispatch queue, which lets you control parallel packet processing.

NS/ARP offload

Your DEXT is directly controlling your hardware so, yes, I think this is something you could configure your hardware to do.

RSC, PTP or packet timestamping and TSN.

I'm not entirely sure how all of those are handled but it might be helpful to look at the NetworkingDriverKit Constants list.

__
Kevin Elliott
DTS Engineering, CoreOS/Hardware

Hi, @DTS Engineer

In the RSS feature of Windows (e.g., as described on the website https://learn.microsoft.com/en-us/windows-hardware/drivers/netcx/rsc-offload), the driver developer uses NET_ADAPTER_RECEIVE_SCALING_HASH_SECRET_KEY to request a hash key from the system. This hash key is then used with an algorithm to calculate a hash value, which determines the specific QueueID.

Then, I reviewed this document IOUserNetworkRxSubmissionQueue/withpool

As I understand it, if I need 4 queues, they can be created using withPool to initialize IOUserNetworkPacketQueueId queueId[0~3]. Each queue extracts packets from the same IOUserNetworkPacketBufferPool, with capacity and bufferCount set as uint32_t capacity, uint32_t bufferCount. However, I am uncertain how to declare or assign values to the two parameters, OSObject * target and DequeueAction dequeueAction. If possible, please provide more information or detailed documentation.

Additionally, are there predefined functionality or value lists for the two parameters void * refCon and IOOptionBits options that can be used for specific feature configurations?

I would like to know the mechanism by which this method determines which QueueID a packet should be placed into or dequeued from for both Tx and Rx. I hope to get a detailed explanation of the exact usage, sample code, and whether this aligns with your statement, "The queue architecture above handles this."

Thanks

First off, I need to be up front on this point:

If possible, please provide more information or detailed documentation.

This isn't directly stated but, practically speaking, but implementing a DriverKit DEXT basically requires a solid understanding of IOKit, as much of DriverKit's actual implementation is directly copied or derived from IOKit. This hasn't been an issue for most DEXT developers because they already had quite deep understanding of IOKit, as they're typically "porting" an existing IOKit KEXT to DriverKit, not actually creating a truly new DEXT from "scratch".

It sounds like you're coming into this as a new developer and I'm afraid that's likely to require a more difficult process of education and indepedant investigation. I can try and provide some broad guidance and direction but, as detailed documentation and sample code are both somewhat limited. However, NetworkingDriverKit does have the sample "Connecting a network driver" built on PCIDriverKit*, which you may have overlooked.

*Keep in mind that a NetworkingDriverKit DEXT typically involves at least two DEXT frameworks, one for the underlying hardware transport (PCI/USB) which is then presented to the networking system through NetworkingDriverKit.

In terms of getting started with this, IOKit has actually been fairly well documented. That documentation is relatively old, however, it's core API has changed very little and it's fundamental architecture hasn't changed at all.

So, the core IOKit documentation sources are:

• IOKit Fundamentals-> This describes the broad architecture and design of IOKit.

• IOKit Device Driver Design Guidelines-> This is a more detailed, practical guide to IOKit an KEXT development. It's not a directly relevant as I/O Kit Fundamentals, but it's still worth understanding.

• Accessing Hardware From Applications-> This covers the user space IOKit API, which are exactly the same APIs DriverKit use.

In addition, much of the kernel and many IOKit families are available in our opensource releases. DriverKit itself not available, but it can be helpful to see how an IOKit KEXT is implemented.

Looking at specifics:

However, I am uncertain how to declare or assign values to the two parameters, OSObject * target

By convention, "target" basicallymeans "the object this message/method is for". There are many places in IOKit where it's necessary to create a C function callback for something that would (ideally) be represented as a C++ method and the "target" method is provided to act as the "this" pointer in those functions.

and DequeueAction dequeueAction.

DequeAction is a function pointer typedef'ed in IOUserNetworkPacketQueue.h. Here is that definition:

typedef uint32_t (*DequeueAction)(OSObject *target,
                                      IOUserNetworkPacketQueue *queue,
                                      IOUserNetworkPacket **packetArray,
                                      uint32_t packetCount, void *refCon);

Additionally, are there predefined functionality or value lists for the two parameters void * refCon

By convention, a "refCon" pointer is an arbitrary value the system accepts from you at some earlier configuration point and then returns to you in each callback. It has no defined type (indeed, it doesn't even need to be a pointer) and is not managed by the system in ANY way. If you want to "attach" some value/data to the callback, you can use the refCon. If you won't want to... then don't.

and IOOptionBits options that can be used for specific feature configurations?

I'm not sure which method your actually looking at here, but "IOOptionBits" is a generic IOKit type used to indicate that a specific value is a bit field. In many places it's included in the function definition to provide for future API evolution, not because any options have actually been defined. As far as I can tell, that true of most the IOOptionBits in NetworkingDriverKt, with the exception of allocatePacket. That method defaults to "kIOUserNetworkNonBlocking" but can also be set to "kIOUserNetworkBlocking".

I'm not sure that this universal, but the default value of "0" like this:

    virtual bool
    initWithPool(IOUserNetworkPacketBufferPool *pool,
                 IOUserNetworkPacketDescriptor *descriptor,
                 IOOptionBits options = 0) LOCALONLY;

...often indicates that not options have been defined. If/when we introduce options in a method that had none, we often do so by defining a new "0" option (which preserves the existing behavior) alongside the new options we're adding, then change the default to new "0" option.

I would like to know the mechanism by which this method determines which QueueID a packet should be placed into or dequeued from for both Tx and Rx. I hope to get a detailed explanation of the exact usage, sample code, and whether this aligns with your statement, "The queue architecture above handles this."

Take a look at "Connecting a network driver" and see if that clarifies things.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

@DTS Engineer

We are indeed developing the PCIe DEXT from scratch, as we do not have any prior KEXT products related to this. I will take the time to thoroughly understand IOKit before consulting you on technical questions.

However, I’d like to start by discussing the concept with you to understand whether the functionality I am aiming for can be achieved on Apple’s platform.

To begin with, let’s assume that I have created four independent queues in the driver: IOUserNetworkTxSubmissionQueue/IOUserNetworkTxCompletionQueue, namely Queue[0], Queue[1], Queue[2], and Queue[3], where Queue[3] has the highest priority, and Queue[0] has the lowest.

Would it be possible for a user, through an application, to determine which specific application-generated packets should be processed and transmitted via one of the driver-created queues [Queue[0] ~ Queue[3]] based on priority? Our goal is to implement functionality on macOS similar to the "Realtek Bandwidth Control" feature on Windows, where the transmission priority of different applications can be managed to achieve traffic shaping by application priority.

The queue architecture above handles this. Every queue has an associated dispatch queue, which lets you control parallel packet processing.

I would like to provide additional information regarding the points mentioned in my previous response. The workflow of RSS (Receive-Side Scaling) on Windows is roughly as follows:

"RSS calculates the queue number for each packet based on a hash function and an indirection table. The upper layer uses this to assign processes to corresponding CPU cores. The hash value is calculated using TCP/UDP ports or IPv4/IPv6 addresses, and the hardware uses hash value bits [6:0] to look up the indirection table to obtain the queue number."

Does "The queue architecture above handles this" imply that the example I previously mentioned—"setting network packet transmission/receiving priorities for applications via an application"—is achievable?

Additionally, in macOS, can a user application transmit a specific value to the hardware device to enable packets from a particular application to be received by a specific Queue ID?

I am doing my best to clearly articulate the functionality we hope to achieve with Multi-queue, and I hope that our understanding of the issue is aligned.

Does "The queue architecture above handles this" imply that the example I previously mentioned—"setting network packet transmission/receiving priorities for applications via an application"—is achievable?

Sort of. In terms of the user space API side, there are a variety of APIs that clients can set to control priority, for example, NSParameter.serviceClass.

However, I'm not sure exactly how that's mapped into the hardware layer, nor am I sure that this kind of logic:

Additionally, in macOS, can a user application transmit a specific value to the hardware device to enable packets from a particular application to be received by a specific Queue ID?

...actually makes sense in our modern network stack. The issue here is that the user space network stack* means that far less of the typical "network stack" is actually in the kernel than would be the case in a standard Unix system (and I suspect Windows as well).

*There's a good overview of this starting ~45 minutes into "Introducing Network.framework: A modern alternative to Sockets" from WWDC 2018.

I think that part of what that architecture changes is that your driver is holding far fewer packets than it would previously have held (because buffered packets are being held in user space instead of the kernel) and that much of the prioritization will be done by user space reordering packets before they're pushed into the kernel. Similarly, the queueing system is primarily about recycling I/O buffers than it is about true "prioritization", as much of the prioritization was done before the packets ever reached your driver.

I am doing my best to clearly articulate the functionality we hope to achieve with Multi-queue, and I hope that our understanding of the issue is aligned.

I think this is case where you're going to need to get driver up and running, then do your own testing and experimentation to work out what works best for your hardware.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

I really appreciate your detailed explanations earlier; I have benefited greatly from them.

I would like to continue exploring the topic in greater depth, focusing on the transmission mechanisms between Tx and Rx. My goal is to understand how data is transferred between DEXT and the upper layers.

[Tx]

In Linux, Qdisc (Queueing Discipline) is responsible for:

• Determining packet transmission priority.
• Controlling packet delay or packet drop policies.
• Implementing traffic shaping and bandwidth limitations.

By using tc (Traffic Control), it is possible to influence how packets are distributed across different Driver Queues. However, the final decision is determined by skb->queue_mapping, which is usually set by Qdisc (such as mq or multiq) or XPS (Transmit Packet Steering).

In macOS DriverKit (DEXT), I would like to ask:

1. Does macOS have a corresponding mechanism that allows developers to manually assign packets to specific TX Queues?

In Linux, ndo_select_queue() allows developers to manually assign packets to a specific TX Queue based on Flow Hashing or QoS policies.

1. Does macOS provide a similar API for developers?
2. Is there a way to manually assign certain types of traffic (e.g., PTP, VoIP, TCP, UDP) to a specific IOUserNetworkTxSubmissionQueue?

The queueing system is primarily about recycling I/O buffers than it is about true 'prioritization,' as much of the prioritization was done before the packets ever reached your driver.

1. Does this imply that macOS’s upper networking stack has already completed CPU load balancing and QoS prioritization before the packet reaches the DriverKit extension (DEXT)?
2. Can it be inferred that macOS has already optimized CPU workload distribution, meaning that even when developing a 10GbE Ethernet driver, using a single TX queue will not create performance bottlenecks?
3. Does this mean that TX Path in macOS is primarily a hardware interaction layer rather than an active network packet scheduling layer?

[Rx]

In Linux, ethtool allows manual configuration of RSS (Receive Side Scaling) via set_rxfh, including:

• Configuring Flow Types (TCP, UDP, IPv4, IPv6).
• Modifying the RSS Indirection Table.
• Changing the RSS Hash Key.

In macOS DriverKit (DEXT), I would like to ask:

1. Can RSS be manually configured during the kern_return_t Start(IOService *Provider) override; stage?
2. Does macOS provide an API that allows developers to modify RSS Flow Type, Indirection Table, or Hash Key?
3. Are there any APIs in macOS similar to Linux’s get_rxfh/set_rxfh for configuring RSS?

The queue architecture above handles this. Every queue has an associated dispatch queue, which lets you control parallel packet processing.

1. Does this mean that IOUserNetworkRxSubmissionQueue / IOUserNetworkRxCompletionQueue inherently manage CPU load balancing, eliminating the need for developers to manually configure RSS Indirection Tables as required in Linux?
2. Can it be confirmed that macOS DEXT only requires a single serial queue to fully support high-performance Ethernet (e.g., 10GbE), and that RSS Hash Key tuning is unnecessary for efficient packet processing?

There are many issues mentioned above, and I hope you can address them one by one. Thank you.

Does macOS have a corresponding mechanism that allows developers to manually assign packets to specific TX Queues?

Yes, the specifics are a bit different than other platforms. In user space, service class can be set through multiple mechanisms, with NWParameters.serviceClass being the most straightforward example. On the DEXT side, IOUserNetworkTxSubmissionQueue.withPoolAndServiceClass allows you to create a submission queue that that matches the service class. The system will then route packets to queues based on that configuration.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware