Creating MLMultiArray from IOSurface backed CVPixelBuffer fixes strides to multiples of 32

Question

Created Aug ’22

Replies 1

Boosts 0

Views 1.7k

Participants 2

Hi,

I have found that when creating a MLMultiArray from an IOSurface backed CVPixelBuffer that the striding across the last dimension gets fixed to a multiple of 32. Say I have some objective-c code that creates a MLMultiArray from a CVPixelBuffer like so;

    CVPixelBufferRef pixelBuffer = NULL;
    NSDictionary* pixelBufferAttributes = @{
        (id)kCVPixelBufferIOSurfacePropertiesKey: @{}
    };
    
    // Since shape == [2, 3, 4], width is 4 (= shape[2]) and height is 6 (= shape[0] * shape[1]).
    CVPixelBufferCreate(kCFAllocatorDefault, 4, 6, kCVPixelFormatType_OneComponent16Half, (__bridge CFDictionaryRef)pixelBufferAttributes, &pixelBuffer);
    MLMultiArray *multiArray = [[MLMultiArray alloc] initWithPixelBuffer:pixelBuffer shape:@[@2, @3, @4]];

This successfully creates a MLMultiArray from an IOSurface backed CVPixelBuffer. If I inspect the MLMultiArray, I see that the shape and strides are the following;

//_shape: [2, 3, 4]
//_strides: [96, 32, 1]

To me it makes more sense that the strides should be;

//_strides: [12,4,1]

I assume that this is likely hardcoded to be multiples of 32 somewhere in the MLMultiArray initialisation code and since strides is a read only member variable of MLMultiArray it cannot be changed explicitly. Is it possible to set the striding explicitly in any way, or create an MLMutliArray from IOSurface backed buffer that allows the striding to be set differently ?

Boost

Answer 1

Frameworks Engineer OP

Apple

Aug ’22

The memory layout (strides) of MLMultiArray is determined by the backing pixel buffer, which often uses 64-bytes alignment per row. Since the scalar type of the MLMultiArray is Float16, the stride of the row is some multiples of 32 scalars. (64 / sizeof(Float16)).

You have some control over the pixel buffer's memory layout (See kCVPixelBufferBytesPerRowAlignmentKey), but it unlikely allows "no-padding" layout because such layout is inefficient or unsupported by the compute units.

0