Apple Silicon M1 crashing with IOPCIFamily based custom KEXT

Question

Created Dec ’20

Replies 8

Boosts 0

Views 3.7k

Participants 8

We have developed an IOPCIFamily based custom KEXT to communicate with Thunderbolt interface storage device.
This KEXT is working fine with Apple machines with Intel CPUs in all types of machines (iMac, iMac Pro and MacBooks).

We tested this KEXT with Apple Silicon M1 machine where we are observing crash for the very first command we send to the Thunderbolt device.

We observed that there is difference in number of bits in Physical Address we use for preparing command PRPs.
In Intel machines we get 28-Bit Physical Address whereas in M1 we are getting 36-Bit address used for PRPs.

We use inTaskWithPhysicalMask api to allocate memory buffer we use for preparing command PRPs.
Below are the options we have used for this:

options: kIOMemoryPhysicallyContiguous | kIODirectionInOut
capacity: 16kb
physicalMask: 0xFFFFF000UL (We want 4kb aligned memory)

According to below documentation, we have to use inTaskWithPhysicalMask api to get memory below 4gb.

https://developer.apple.com/library/archive/documentation/Darwin/Conceptual/64bitPorting/KernelExtensionsandDrivers/KernelExtensionsandDrivers.html#//apple_ref/doc/uid/TP40001064-CH227-SW1

Some devices can only handle physical addresses that fit into 32 bits. To the extent that it is possible to use 64-bit addresses you should do so, but for these devices, you can either use IODMACommand or the initWithPhysicalMask method of IOBufferMemoryDescriptor to allocate a bounce buffer within the bottom 4 GB of physical memory.

So just want to know what's the difference between Intel and ARM64 architecture with respect to physical memory access.
Is there any difference between byte order for physical memory address..??

Crash log is given below:

panic(cpu 0 caller 0xfffffe0016e08cd8): "apciec[0:pcic0-bridge]::handleInterrupt: Request address is greater than 32 bits linksts=0x99000001 pcielint=0x00020000 linkcdmsts=0x00000800 (ltssm 0x11=L0)\n"
Debugger message: panic
Memory ID: 0x6
OS release type: User
OS version: 20C69
Kernel version: Darwin Kernel Version 20.2.0: Wed Dec 2 20:40:21 PST 2020; root:xnu-7195.60.75~1/RELEASEARM64T8101
Fileset Kernelcache UUID: 3E6AA74DF723BCB886499A5AAB34FA34
Kernel UUID: 48F71DB3-6C91-3E62-9576-3A1DCEF2B536
iBoot version: iBoot-6723.61.3
secure boot?: YES
Paniclog version: 13
KernelCache slide: 0x000000000dbfc000
KernelCache base: 0xfffffe0014c00000
Kernel slide: 0x000000000e73c000
Kernel text base: 0xfffffe0015740000
Kernel text exec base: 0xfffffe0015808000
machabsolutetime: 0x12643a9c5
Epoch Time: sec usec
Boot : 0x5fe06736 0x0009afbc
Sleep : 0x00000000 0x00000000
Wake : 0x00000000 0x00000000
Calendar: 0x5fe067fd 0x0006569d

CORE 0 recently retired instr at 0xfffffe0015971798
CORE 1 recently retired instr at 0xfffffe0015972c5c
CORE 2 recently retired instr at 0xfffffe0015972c5c
CORE 3 recently retired instr at 0xfffffe0015972c5c
CORE 4 recently retired instr at 0xfffffe0015972c60
CORE 5 recently retired instr at 0xfffffe0015972c60
CORE 6 recently retired instr at 0xfffffe0015972c60
CORE 7 recently retired instr at 0xfffffe0015972c60
Panicked task 0xfffffe166ce9e550: 75145 pages, 462 threads: pid 0: kernel_task
Panicked thread: 0xfffffe166d053918, backtrace: 0xfffffe306cb4b6d0, tid: 141
lr: 0xfffffe0015855f8c fp: 0xfffffe306cb4b740
lr: 0xfffffe0015855d58 fp: 0xfffffe306cb4b7b0
lr: 0xfffffe0015977f5c fp: 0xfffffe306cb4b7d0
lr: 0xfffffe0015969914 fp: 0xfffffe306cb4b880
lr: 0xfffffe001580f7e8 fp: 0xfffffe306cb4b890
lr: 0xfffffe00158559e8 fp: 0xfffffe306cb4bc20
lr: 0xfffffe00158559e8 fp: 0xfffffe306cb4bc90
lr: 0xfffffe0015ff03f8 fp: 0xfffffe306cb4bcb0
lr: 0xfffffe0016e08cd8 fp: 0xfffffe306cb4bd60
lr: 0xfffffe00166bc778 fp: 0xfffffe306cb4be30
lr: 0xfffffe0015f2226c fp: 0xfffffe306cb4be80
lr: 0xfffffe0015f1e2f4 fp: 0xfffffe306cb4bec0
lr: 0xfffffe0015f1f050 fp: 0xfffffe306cb4bf00
lr: 0xfffffe0015818c14 fp: 0x0000000000000000
Kernel Extensions in backtrace:
com.apple.driver.AppleEmbeddedPCIE(1.0)[4F37F34B-EE1B-3282-BD8B-00009B954483]@0xfffffe00166b4000->0xfffffe00166c7fff
dependency: com.apple.driver.AppleARMPlatform(1.0.2)[5CBA9CD0-E248-38E3-94E5-4CC5EAB96DE1]@0xfffffe0016148000->0xfffffe0016193fff
dependency: com.apple.driver.IODARTFamily(1)[88B19766-4B19-3106-8ACE-EC29201F00A3]@0xfffffe0017890000->0xfffffe00178a3fff
dependency: com.apple.iokit.IOPCIFamily(2.9)[5187699D-1DDC-3763-934C-1C4896310225]@0xfffffe0017c48000->0xfffffe0017c63fff
dependency: com.apple.iokit.IOReportFamily(47)[93EC9828-1413-3458-A6B2-DBB3E24540AE]@0xfffffe0017c64000->0xfffffe0017c67fff
com.apple.driver.AppleT8103PCIeC(1.0)[35AEB73B-D51E-3339-AB5B-50AC78740FB8]@0xfffffe0016e04000->0xfffffe0016e13fff
dependency: com.apple.driver.AppleARMPlatform(1.0.2)[5CBA9CD0-E248-38E3-94E5-4CC5EAB96DE1]@0xfffffe0016148000->0xfffffe0016193fff
dependency: com.apple.driver.AppleEmbeddedPCIE(1)[4F37F34B-EE1B-3282-BD8B-00009B954483]@0xfffffe00166b4000->0xfffffe00166c7fff
dependency: com.apple.driver.ApplePIODMA(1)[A8EFA5BD-B11D-3A84-ACBD-6DB25DBCD817]@0xfffffe0016b0c000->0xfffffe0016b13fff
dependency: com.apple.iokit.IOPCIFamily(2.9)[5187699D-1DDC-3763-934C-1C4896310225]@0xfffffe0017c48000->0xfffffe0017c63fff
dependency: com.apple.iokit.IOReportFamily(47)[93EC9828-1413-3458-A6B2-DBB3E24540AE]@0xfffffe0017c64000->0xfffffe0017c67fff
dependency: com.apple.iokit.IOThunderboltFamily(9.3.2)[11617399-2987-322D-85B6-EF2F1AD4A794]@0xfffffe0017d80000->0xfffffe0017e93fff

Stackshot Succeeded Bytes Traced 277390 (Uncompressed 703968) **

System Information:
Apple Silicon M1
BigSur 11.1
Model: Macmini9,1

Any help or suggestion is really appreciated.
Thanks

Answer 1

stang2021 OP

Jan ’21

Did you try setPreparationID to your memory descriptor?

Answer 2

PCBuff OP

Jan ’21

I am having a similar issue with IOBufferMemoryDescriptor::inTaskWithPhysicalMask.

It seems like inTaskWithPhysicalMask does not allow the physicalMask to restrict the output address to only 32-bits on an M1 based mac. If I change the physicalMask to be 0xFFFFFF000 (>32 bits) then the inTaskWithPhysicalMask succeeds but now the hardware has an issue with the 64-bit addressing.

Also, it would seem that setPreparationID is for user mode code, not Kernel.

Have you found a way round this?

Answer 3

sudhanshu11feb OP

Feb ’21

Yes..
I have found the solution. Use IODMACommand .

Answer 4

PCBuff OP

Feb ’21

Care to share some example source on how you were able to do that?

I am also using IODMACommand. However if the IOBufferMemoryDescriptor::inTaskWithPhysicalMask has the physicalMask set to 0xFFFFF000 (32 bits) then inTaskWithPhysicalMask fails and it wouldn't matter what IODMACommand does since you can't get there.

Answer 5

ywmac OP

Feb ’26

Hello, may I ask if your driver is a DMA driver for PCIe+FPGA? If so, will using a 64 bit IoVM address for DMA transfer be intercepted by the PCIe root port of the M1 Mac?

Answer 6

DTS Engineer OP

Apple

Feb ’26

Scratch my old advice advice, I clearly wasn't thinking clearly yesterday:

Try with 0xFFFFFFFFFFFFF000ULL parameter.

~~In 64 bit, "mach_vm_address_t physicalMask" will be a 64 bit value, so the compiler will "helpfully" zero extend a mask of (spaces added for clarity):~~

Correcting that:

The correct mask for a 32 bit address would be "0x00000000FFFFFFFFULL".
The modern DART architecture makes this largely irrelevant, as IODMACommand is what gets you an I/O Address (usable on the PCI bus). I'm still looking into this, but at this point I don't think the physical mask API is actually all that useful/relevant.
It's possible we're actually blocking the old 32-bit mask (as a way of catching errors) but I'm still looking into that.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Answer 7

mxweas OP

2w

@DTS Engineer any updates here?

Answer 8

DTS Engineer OP

Apple

2w

@DTS Engineer, any updates here?

Hmm... Looks like this slipped off my radar. In any case, if you're talking about this:

I am also using IODMACommand. However, if the IOBufferMemoryDescriptor::inTaskWithPhysicalMask has the physicalMask set to 0xFFFFF000 (32 bits), then inTaskWithPhysicalMask fails, and it wouldn't matter what IODMACommand does since you can't get there.

Then the answer is "Yes, that doesn't work, don't do that".

That is, drop the call to IOBufferMemoryDescriptor::inTaskWithPhysicalMask entirely and just let IODMACommand's constraints give you the bus address you actually need/want.

In terms of what's going on here, the problem here is that we're used to thinking in terms of two addresses (VM and physical), but there are actually 3 address types:

VM addresses -> used to actually manipulate data "in memory".
Physical addresses -> the location data is stored in true physical memory.
Bus Addresses -> Addresses used to perform DMA commands on the PCI bus.

The problem here is that Intel has a global mapper, which means that, in many cases, you can use 2 & 3 somewhat interchangeably. Apple Silicon doesn't work that way. More specifically:

Apple Silicon does NOT have a global mapper. Instead, each PCI target is basically maintaining its own DART (Device Address Relocation Table), which generates bus addresses which are ONLY valid on that particular bus.
Apple Silicon physical memory architecture "starts" at 4G, so it CANNOT generate 32-bit physical addresses (type #2). I believe this was at least partly done to better "catch" drivers that were not using IODMACommand.

That second point is why "IOBufferMemoryDescriptor::inTaskWithPhysicalMask“ failed - you asked the system for a 32-bit physical address and it can't do that, as it doesn't actually have any addressable memory in that range. However, that doesn't actually matter as you don't actually need/want a physical address at all. You want a 32-bit bus address, which IODMACommand should be able to provide.

And, yes, this means that true physical addresses are effectively useless on Apple Silicon, as they're only relevant to the VM system itself which you're, presumably, not part of.

On another note here is that, in practice, address segmentation does NOT actually occur on Apple Silicon, even when dealing with very large chunks of memory. This is hinted at here in "Address Translation on 64-Bit System Architectures":

"One side effect of these changes in the memory subsystem is that OS X is likely to return physically contiguous page ranges in memory regions. In earlier versions of OS X, the system returned multi-page memory regions in reverse order, beginning with the last page and moving towards the first page. Because of this, a multi-page memory region seldom contained a physically contiguous range of pages."

However, the effect is much more "pronounced" on Apple Silicon. More specifically, if your provider chain leads to a PCI device (this is true for all physical devices), then IODMACommands segment generation will return a single segment for basically "all" IOMemoryDescriptors, up to the point at which memory descriptor allocation itself starts to fail [1].

The segment infrastructure continues to exist, but that’s essentially an accident of API history, not something the system actually "uses". Notably, SCSIControllerDriverKit's fBufferIOVMAddr usage assumes that segmentation will not occur, and many of our drivers actually fail and even assert() (panic'ing the kernel) if segmentation occurs.

[1] Limits have been hard-coded in a number of places, but those limits are WELL above (think several GB) what would be considered normal/sensible. More to the point, you'll fail at descriptor creation instead of generating multiple segments.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware