Metal texture / CGImage from data asset

I believe native reading of OpenEXR format is (at least officially on macOS) supported both on recent macOS and iOS versions:
https://forums.developer.apple.com/thread/97119

I'd like to load an OpenEXR image to a Metal texture, probably via MTKTextureLoader.newTexture().


My problem is that XCode doesn't recognise OpenEXR files as texture assets, but as data asset.


This means I cannot use MTKTextureLoader.newTexture(name: textureName, ...).

What cross platform (recent macOS / iOS) options are there to read an image from a data asset?

Since .newTexture supports CGImage, I'd guess that the natural way would be to load into CGImage, but I don't quite understand how.


Or should I simply make an URL out of the data asset's file and try to load that one?

Replies

The easy answer for trivial cases is to use the CGImageSourceRef:

Code Block C
int main(int argc, const char * argv[]) {
    
    int error = 0;
    id <MTLTexture> tex = MakeMipMapEXRTexture(argc, argv, & error);
   
   
    // load arg1 as the path to the EXR file
    CFStringRef s = CFStringCreateWithCString(NULL, argv[1], kCFStringEncodingUTF8 );
    CFURLRef url = CFURLCreateWithFileSystemPath(NULL, s, kCFURLPOSIXPathStyle, false);
    CFRelease(s);
        
    // Create a CGImageSource and make a CGImage out of it
    CGImageSourceRef source = CGImageSourceCreateWithURL( url, NULL);
    CFRelease(url);
    CGImageRef image = CGImageSourceCreateImageAtIndex(source, 0, NULL);
    CFRelease(source);
    // Create a RGBAf16 context and draw the image into it
    CGContextRef context = CGBitmapContextCreate( NULL,
                                                   CGImageGetWidth(image),
                                                   CGImageGetHeight(image),
                                                   16,
                                                   CGImageGetWidth(image) * 8,
                                                   CGColorSpaceCreateWithName( kCGColorSpaceSRGB ),
                                                   kCGBitmapByteOrder16Host | kCGImageAlphaPremultipliedLast | kCGBitmapFloatComponents );
    CGRect where = CGRectMake(0, 0, CGImageGetWidth(image), CGImageGetHeight(image));
    CGContextClearRect( context, where);
    CGContextDrawImage( context, where, image);
    CGContextFlush(context);
    unsigned long width = CGImageGetWidth(image);
    unsigned long height = CGImageGetHeight(image);
    void * bytes = CGBitmapContextGetData(context);
    size_t rowBytes = CGBitmapContextGetBytesPerRow(context);
    
    CGImageRelease(image);
    
    @autoreleasepool {
        id <MTLDevice> device = MTLCreateSystemDefaultDevice();
        
        MTLTextureDescriptor * d = [MTLTextureDescriptor texture2DDescriptorWithPixelFormat: MTLPixelFormatRGBA16Float
                                                                                      width: width
                                                                                     height: height
                                                                                  mipmapped: NO];
        id <MTLTexture> tex = [device newTextureWithDescriptor: d];
        [tex replaceRegion: (MTLRegion){ {0,0,0}, {width, height, 1}}
               mipmapLevel: 0
                 withBytes: bytes
               bytesPerRow: rowBytes];
    }
   
  CGContextRelease(context);
   
    return 0;
}


That said, this isn't going to get you very far with complex cases like cube maps, ripmaps/mipmaps and depth buffers, which are also representable in OpenEXR. So there is that. You also need to pay attention to color with OpenEXR. It is encoded in linear gamma, defined by a set of chromaticities, and not something necessarily simple like SRGB. You have to actually look at the chromaticities, for example to distinguish between sRGB, another RGB or XYZ, and sometimes the data is YCbCr which is another level of stuff. In so far as colorspace conversions go, if your drawing pipeline is in linear gamma already, then great! Otherwise, you may find yourself having to do some color conversions so that the artwork in your assets aren't completely off. 
If you are tasked with supporting the Full Monte, I recommend using instead AppleEXR.dylib and vImage.  The pathway for mipmaps with full colorspace conversion looks something like this:

PART 1:

Code Block C
id <MTLTexture> MakeMipMapEXRTexture( int argc, const char *argv[], int * e )
{
    int error = 0;
    const axr_flags_t myFlags =
#if DEBUG
        axr_flags_print_debug_info;
#else
        axr_flags_default;
#endif
void * fileData = mmap(...);
    axr_error_t axrErr = axr_error_success;
    
    // Note: in C many of the function arguments below have default values and can be omitted
    
    // create a axr_data_t to represent the file. See also axr_introspect_data()
    axr_data_t axrData = axr_data_create( fileData, fileSize, &axrErr, myFlags,
                                         ^( void * nonnull fileData, size_t fileSize ){ munmap( fileData, fileSize);});
    if( NULL == axrData )
    {
        *e = (int) axrErr;
        return nil;
    }
    
    // Look for the layer and part that I want in the file
    // Each EXR file may be segmented up into many parts
    unsigned long desiredPartition = 0;
    const char * desiredLayerName = NULL;
    unsigned long partitionCount = axr_data_get_part_count(axrData);
  unsigned long mipLevelCount = 0;
    bool found = false;
    for( unsigned long part = 0; part < partitionCount && ! found; part )
    {
        axr_part_info_t partInfo = axr_data_get_part_info( axrData,  part, axr_part_info_current);
       
      // and each part may have many layers!!
        unsigned long layerCount = axr_data_get_layer_count( axrData, part);
        for( unsigned long layer = 0; layer < layerCount; layer)
        {
            axr_layer_info_t layerInfo = axr_data_get_layer_info( axrData, part, layer, axr_layer_info_current);
            
            if( false == IsThisTheLayerToDisplay( &partInfo, &layerInfo ))
                continue;
          desiredPartition = part;
            desiredLayerName = layerInfo.name;
            mipLevelCount = axr_data_get_level_count( axrData, part);
            found = true;
            break;
        }
    }
 
    // Figure out how big to make the buffer to receive the pixels
    axr_decoder_t tempDecoder = axr_decoder_create_rgba( axrData, desiredLayerName, desiredPartition, 0, myFlags);
    axr_type_t sampleType = axr_decoder_get_channel_info(tempDecoder, 0, axr_channel_info_current).sampleType;
    uint32_t channelSize = (uint32_t) axr_type_get_size(sampleType);
    uint32_t pixelSize = 4 * channelSize;       // RGBA = 4 channels
    axr_decoder_info_t info = axr_decoder_get_info(tempDecoder, axr_decoder_info_current);
    vImage_Buffer buf;
    vImage_Error vImageError = vImageBuffer_Init( &buf, info.subregion.size.height, info.subregion.size.width, pixelSize * 8 /*bits per byte */, kvImageNoFlags);
    if( NULL == buf.data )
    {
           // ...handle malloc failure ...
    }
   
    // read the desired content from the file into buf as RGBA pixels
    id <MTLDevice> device = MTLCreateSystemDefaultDevice();
    MTLTextureDescriptor * desc = [MTLTextureDescriptor texture2DDescriptorWithPixelFormat: channelSize == 2 ? MTLPixelFormatRGBA16Float : MTLPixelFormatRGBA32Float
                                                                                     width: buf.width
                                                                                    height: buf.height
                                                                                 mipmapped: mipLevelCount > 1];
    // don't forget an autoreleasepool when working with Metal, especially command buffers!!
    id <MTLTexture> tex = [device newTextureWithDescriptor: desc];
#if ! has_feature(objc_arc)
    [desc release];
#endif
    // sort out colorspace
    CGColorSpaceRef exrColorSpace = axr_decoder_create_rgba_colorspace( tempDecoder, myFlags, NULL);
    CGColorSpaceRef desiredColorSpace = CGColorSpaceCreateWithName( kCGColorSpaceExtendedSRGB );        // or whatever colorspace you are using for your rendering surface in Metal
  // Caution: extended linear HDR to SDR colorspace conversions on BigSur don't do tone mapping yet,
    //          so large out of [0,1] values would get clamped to [0,1] possibly causing noticeable hue shifts.
  //   I used kCGColorSpaceExtendedSRGB so this wouldn't happen. 
    //   EXR files may put values > 1 or < 0 into your shader.
    
#if ! has_feature(objc_arc)
    os_release(tempDecoder);
#endif

<continued...>
PART 2:

Code Block C
  // make a colorspace conversion recipe
  CGColorConversionInfoRef recipe = CGColorConversionInfoCreateWithOptions( exrColorSpace, desiredColorSpace,  NULL /* options */ );
    vImage_CGImageFormat exrFormat = (vImage_CGImageFormat)
    {
        .bitsPerPixel = pixelSize * 8,
        .bitsPerComponent = channelSize * 8,
        .colorSpace = exrColorSpace,
        .bitmapInfo = kCGBitmapFloatComponents | kCGImageAlphaLast | (channelSize == 2 ? kCGBitmapByteOrder16Host : kCGBitmapByteOrder32Host)
    };
    vImage_CGImageFormat textureFormat = (vImage_CGImageFormat)
    {
        .bitsPerPixel = pixelSize * 8,
      .bitsPerComponent = channelSize * 8,
      .colorSpace = desiredColorSpace,
      .bitmapInfo = kCGBitmapFloatComponents | kCGImageAlphaLast | (channelSize == 2 ? kCGBitmapByteOrder16Host : kCGBitmapByteOrder32Host)
    };
    vImage_Flags vImageFlags =
#if DEBUG
    kvImagePrintDiagnosticsToConsole;
#else
    kvImageNoFlags;
#endif
    vImageConverterRef converter = vImageConverter_CreateWithCGColorConversionInfo( recipe, &exrFormat, &textureFormat, NULL, vImageFlags, &vImageError);
    vImage_Buffer converted = buf;
    if( converter && kvImageNoError != vImageConverter_MustOperateOutOfPlace( converter, &buf, &converted, kvImageNoFlags))
    {
        vImageBuffer_Init( &converted, buf.height, buf.width,  pixelSize * 8, kvImageNoFlags);
     
        if( NULL == buf.data)
        {
            // handle malloc failure
        }
    }
    if( tex )
        for( unsigned long mipLevel = 0; mipLevel < mipLevelCount; mipLevel )
        {
            // Make a decoder for the part of the file you want to read
            //    default parameters will read the default layer from the first part
            //  axr_decoder_create_rgba is a simplified interface for RGBA data. If you are after depth information,
            //  you'll possibly want to take the longer route with axr_decoder_create() and configure manually.
            axr_decoder_t decoder = axr_decoder_create_rgba( axrData, desiredLayerName, desiredPartition, mipLevel, myFlags);
            if( NULL == decoder )
            {
        #if ! has_feature(objc_arc)
                os_release(axrData);
                [tex release];
        #endif
                vImageConverter_Release(converter);
                free(buf.data);
                *e = -1;
                return nil;
            }
            axr_decoder_info_t info = axr_decoder_get_info(tempDecoder, axr_decoder_info_current);
            unsigned long width = info.subregion.size.width;
            unsigned long height = info.subregion.size.height;
            if( axr_error_success == (axrErr = axr_decoder_read_rgba_pixels( decoder, buf.data, buf.rowBytes, 1.0, myFlags )))
            {
                if( converter )
                {
buf.width = converted.width = width; buf.height = converted.height = height;
                    // Note that a similar colorspace conversion is also available on the GPU using MPSImageConversion
                    vImageError = vImageConvert_AnyToAny( converter, &buf, &converted, NULL, kvImageNoFlags);
                    // handle error...
                }
                [tex replaceRegion: (MTLRegion){{0,0,0}, {width, height, 1}}
                       mipmapLevel: mipLevel
                             slice: 0
                         withBytes: converted.data
                       bytesPerRow: converted.rowBytes
                     bytesPerImage: height * converted.rowBytes ];
            }
#if ! has_feature(objc_arc)
            os_release(decoder);
#endif
        }
    vImageConverter_Release(converter);
    if( converted.data != buf.data )
        free(converted.data);
    free(buf.data);
#if ! has_feature(objc_arc)
    os_release(axrData);
#endif
    return tex;
}


AppleEXR is new in Big Sur and associated iOS / iPadOS / tvOS / watchOS releases.