Building Only MSI-Capable Thunderbolt Devices

Apple strongly recommends building only MSI-capable Thunderbolt devices. Ensure your OS X drivers enable MSI when supporting Thunderbolt devices. To ease development, the Mac Pro computers have PCIe slots that support MSI.

OS X services all interrupts on CPU 0 and uses the secondary interrupt context threads to defer work to task-level interrupts rather than primary-interrupt levels. This method ensures that multiple drivers can run in parallel on the available CPUs, that the OS can schedule real-time threads with accuracy, and that the system remains responsive to user interaction. The best practice methods, described in IOKit Fundamentals, encourage the use of the IOWorkLoop abstraction model for device drivers in order to defer work to the IOWorkLoop thread. There may be a small number of devices that require work to be done in the primary interrupt context, however, drivers should spend the least amount of time possible in the primary-interrupt context.

Handling Hardware Exceptions for MSI Support

The hardware should support MSI, but in the unlikely event the hardware device does not support MSI, the following three scenarios are most likely to occur. These three scenarios usually occur in the driver’s filter routine and it is important to understand the ramifications of the results returned from the filter routine:

1. The filter is called, but the device has not signaled an interrupt.

   This may be due to another device sharing the same interrupt pin. The filter routine should return false.

2. The filter is called and the device has signaled an interrupt.

   The driver cannot mask the interrupt source, nor can it handle the interrupt at this level for locking or performance reasons. The filter routine should return true.

3. The filter is called and the device has signaled an interrupt.

   1. The driver can handle the interrupt at this level and return the device to a non interrupting state. If the interrupt action needs to be run, the filter routine should call signalInterrupt().

   2. The driver cannot handle the interrupt at this level, but it can prevent the device from signaling another interrupt (for example, by manipulating an internal mask).

      The filter routine should prevent the device from signaling another interrupt and then call signalInterrupt() to cause the interrupt action to be run. The interrupt action should handle the interrupt and return the device to a state where it can again signal an interrupt. The filter routine should return false.

Apple provides developers with tools that can track primary interrupt times and help developers to minimize the time needed for primary interrupts.

int    index  = 0;

int    source = 0;

for ( index = 0; ; index++ )

{

         IOReturn result      = kIOReturnSuccess;

         int interruptType    = 0;

         result = provider->getInterruptType ( index, &interruptType );

         if ( result != kIOReturnSuccess )

               break;

         if ( interruptType & kIOInterruptTypePCIMessaged )

        {

            source = index;

            break;

        }

}

Using Hot Plug Operation with PCI Devices

PCI device drivers are typically developed with the expectation that the device will not be removed from the PCI bus during its operation. However, Thunderbolt technology allows PCI data to be tunneled through a Thunderbolt connection, and the Thunderbolt cables may be unplugged from the host or device at any time. Therefore, the system must be able to cope with the removal of PCI devices by the user at any time.

The user may freely unplug Thunderbolt devices at any time, with the exception of storage devices, and should be able to sleep and wake the system with devices attached without causing any problems. For all devices, the user must not be able to hang the system (computer) by unplugging a device or cable.

The PCI device drivers used with Thunderbolt devices may need to be updated in order to handle surprise or unplanned removal. In particular, MMIO cycles and PCI Configuration accesses require special attention. When a PCI device that is connected to a Thunderbolt port is detached from the system, the PCIe Root Port must time out any outstanding transactions sent to the device, terminate the transaction as though an Unsupported Request occurred on the bus, and return a value of 0xFFFFFFFF. The Root Port has a completion timeout value that is many milliseconds long and varies depending on the system layout. Real-time scheduling, particularly for audio and video threads, can be affected by transactions that must be timed out.

As a basic guideline, developers should modify their drivers to handle a return value of 0xFFFFFFFF. If any thread, callback, interrupt filter, or code path in a driver receives 0xFFFFFFFF indicating the device has been unplugged, then all threads, callbacks, interrupt filters, interrupt handlers, and other code paths in that driver must cease MMIO reads and writes immediately and prepare for termination.

If 0xFFFFFFFF is a legal value for a particular register offset, one additional read of a different register, which is known to never return 0xFFFFFFFF is the preferred mechanism for determining if the device is still connected. Finally, if I/O Kit has already performed termination and called the driver’s willTerminate() method, no further accesses should be performed.

Once it has been determined that a device is no longer connected, do not try to clean up or reset the hardware as attempts to communicate with the hardware may lead to further delays.

Apple recommends auditing usage of MMIO writes when no further access should be performed. MMIO writes are posted transactions and it is possible for device drivers to queue up multiple writes in a row.

A typical way for a developer to solve this problem is to provide a single bottleneck routine for all MMIO reads and have that routine check the status of the device before beginning the actual transaction. An example of such a routine using little endian fields for its memory mapped apertures follows:

Listing 3-2  Single Bottleneck Routine (MMIO Read/Write)

class AppleSamplePCI

{

      ...

      bool                     fDeviceRemoved;

      volatile unit8_t *       fBaseAddressRegister;

      ...

      unit32_t        ReadRegister ( uint32_t  offset );

      virtual bool    willTerminate ( IOService * provider, IOOptionBits options );

};

unit32_t

AppleSamplePCI::ReadRegister ( unint32_t offset )

{

         unint32_t   result = 0xFFFFFFFF;

         if  ( !fDeviceRemoved )

         {

                result = OSReadLittleInt32 ( fBaseAddressRegister, offset );

                if ( result == 0xFFFFFFFF )

                    fDeviceRemoved = true;

         }

         return result;

}

bool

AppleSamplePCI::willTerminate ( IOService * provider, IOOptionBits options )

{

         fDeviceRemoved = true;

         return super::willTerminate ( provider, options );

}

Similar routines can be written for PCI Configuration cycle transactions, which may receive a return value of 0xFFFFFFFF and MMIO reads of smaller sizes.

<key>IOPCIPauseCompatible</key>

<true/>

static const IOPMPowerState powerStates[kYourDriverPowerStateCount] = {

    // version, capabilityFlags, outputPowerCharacter,

    // inputPowerRequirement, staticPower, unbudgetedPower

    // powerToAttain timeToAttain settleUpTime

    // timeToLower settleDownTime powerDomainBudget

    // Device off; inputPowerRequirement is 0,

    // which matches the flags for kIOPCIDeviceOffState.

    { kIOPMPowerStateVersion1, 0, 0,

      0, 0, 0,

      0, 0, 0,

      0, 0, 0 },

    // Sleep mode; inputPowerRequirement has kIOPMSoftSleep flag set,

    // which matches the flags for kIOPCIDeviceDozeState.

    { kIOPMPowerStateVersion1, 0, kIOPMSoftSleep,

      kIOPMSoftSleep, 0, 0,

      0, 0, 0,

      0, 0, 0 },

    // Device paused; inputPowerRequirement has kIOPMConfigRetained flag set,

    // which matches the flags for kIOPCIDevicePausedState.

    { kIOPMPowerStateVersion1, kIOPMConfigRetained, kIOPMConfigRetained,

      kIOPMConfigRetained, 0, 0,

      0, 0, 0,

      0, 0, 0 },

    // Device active; inputPowerRequirement has kIOPMPowerOn flag set,

    // which matches the flags for kIOPCIDeviceOnState.

    { kIOPMPowerStateVersion1, kIOPMPowerOn | kIOPMUsable, kIOPMPowerOn,

      kIOPMPowerOn, 0, 0,

      0, 0, 0,

      0, 0, 0 }

};