CoreMediaIO Camera Extension: custom properties?

I struggle to add custom properties to my streams as described in the WWDC22 video https://developer.apple.com/videos/play/wwdc2022/10022/ minute 28:17

The speaker describes using this technique in his CIFilterCam demo (would the source code be available please?) to let the app control which filter the extension should apply.

Presumably, there's thus a way to: 1 - define a custom property in the camera extension's stream/device/provider? 2 - be able to use CoreMediaIO (from Swift?) in the app in order to set values of that custom property.

This is not documented anywhere I could find.

Help and sample code would be greatly appreciated.

Thank you. Laurent

Replies

ObjC is what I haz. I believe you must first expose the key via the availableProperties method on your CMIOExtensionStreamSource.

From the header:

typedef NSString *CMIOExtensionProperty NS_TYPED_ENUM API_AVAILABLE(macos(12.3));

From the sample/template code:

- (NSSet<CMIOExtensionProperty> *)availableProperties {
//    return [NSSet setWithObjects:, CMIOExtensionPropertyStreamFrameDuration, nil];
CMIOExtensionProperty myCustomPropertyKey = @"4cc_cust_glob_0000";
    return [NSSet setWithObjects:
				CMIOExtensionPropertyStreamActiveFormatIndex,
				myCustomPropertyKey,
				nil];
}

In the client application you use the CoreMedia C API to set/get the property value:

CMIOObjectPropertyAddress myCustomPropertyAddress = {
	.Selector = FOUR_CHAR_CODE('cust')
	.Scope = kCMI0ObjectPropertyScopeGlobal,
	.mElement = kCMI0ObjectPropertyElementMain };

// CMIOObjectHasProperty(object,&myCustomPropertyAddress);
// CMIOObjectGetPropertyData(object, &myCustomPropertyAddress, 0, NULL, *propertyDataSize, propertyDataSize, propertyValue));

To see how to query the device list, open streams and retrieve property values, this sample may be useful:

https://github.com/bangnoise/cmiotest

When I attempt to access my custom property (on a stream), I get this error:

CMIO_DAL_CMIOExtension_Stream.mm:1165:GetPropertyData 50 wrong 4cc format for key 4cc_cust_glob_0000
CMIO_DAL_CMIOExtension_Stream.mm:1171:GetPropertyData unknown property error cust
CMIOHardware.cpp:328:CMIOObjectGetPropertyData Error: 2003332927, failed

This message is only triggered if I attempt to access the property from the client side. However, the os_log output shows the correct value for the property.

Error code 2003332927 is FOUR_CHAR_CODE('who?') which maps to kCMIOHardwareUnknownPropertyError

I have determined experimentally that under Objective-C (at least), the format of the key for a custom property is as follows:

CMIOExtensionProperty myCustomPropertyKey = @"cust_glob_0000";

The fcc_ prefix is either Swift-specific or superfluous. I'm on 12.3.

Still no data returned, the value I get is always 0.

I must recant my last comment. I was successful in retrieving custom properties on both Device and Stream using the following code:

const CMIOExtensionProperty CMIOExtensionPropertyCustomPropertyData_just = @"4cc_just_glob_0000";
const CMIOExtensionProperty CMIOExtensionPropertyCustomPropertyData_dust = @"4cc_dust_glob_0000";
const CMIOExtensionProperty CMIOExtensionPropertyCustomPropertyData_rust = @"4cc_rust_glob_0000";

+ (NSMutableDictionary*)sampleCustomPropertiesWithPropertySet:(NSSet<CMIOExtensionProperty>*)properties {
NSMutableDictionary* dictionary = [NSMutableDictionary dictionary];
	if ([properties containsObject:CMIOExtensionPropertyCustomPropertyData_just]) {
	CMIOExtensionPropertyState* propertyState = [CMIOExtensionPropertyState propertyStateWithValue:@"Property value for 'just'"];
		[dictionary setValue:propertyState forKey:CMIOExtensionPropertyCustomPropertyData_just];
		}
	if ([properties containsObject:CMIOExtensionPropertyCustomPropertyData_dust]) {
	const size_t sData_length = 12;
	static const unsigned char sData[sData_length] = { 0xFE,0xED,0xFE,0xED, 0xFE,0xED,0xFE,0xED, 0xFE,0xED,0xFE,0xED };
	CMIOExtensionPropertyState* propertyState = [CMIOExtensionPropertyState propertyStateWithValue:[NSData dataWithBytes:sData length:sData_length]];
		[dictionary setValue:propertyState forKey:CMIOExtensionPropertyCustomPropertyData_dust];
		}
	if ([properties containsObject:CMIOExtensionPropertyCustomPropertyData_rust]) {
	NSString* propertyValue = [NSString stringWithFormat:@"Custom property value for property '%@'.", CMIOExtensionPropertyCustomPropertyData_rust ];
	CMIOExtensionPropertyState* propertyState = [CMIOExtensionPropertyState propertyStateWithValue:propertyValue];
		[dictionary setValue:propertyState forKey:CMIOExtensionPropertyCustomPropertyData_rust];
		}
	return dictionary;
}

Then for CMIOExtensionDeviceSource, I passed the custom property values as a dictionary to the initializer:

- (NSSet<CMIOExtensionProperty> *)availableProperties {
    return [NSSet setWithObjects:
			CMIOExtensionPropertyDeviceTransportType,
			CMIOExtensionPropertyDeviceModel,
			CMIOExtensionPropertyCustomPropertyData_just,
			CMIOExtensionPropertyCustomPropertyData_dust,
			CMIOExtensionPropertyCustomPropertyData_rust,
			nil];
}

- (nullable CMIOExtensionDeviceProperties *)devicePropertiesForProperties:(NSSet<CMIOExtensionProperty> *)properties
																	error:(NSError * _Nullable *)outError {
NSMutableDictionary* dictionary = [self sampleCustomPropertiesWithPropertySet:properties];
CMIOExtensionDeviceProperties *deviceProperties = [CMIOExtensionDeviceProperties devicePropertiesWithDictionary:dictionary];

And for CMIOExtensionStreamSource:

- (NSSet<CMIOExtensionProperty> *)availableProperties {
    return [NSSet setWithObjects:
						CMIOExtensionPropertyStreamActiveFormatIndex,
						CMIOExtensionPropertyStreamFrameDuration,
						CMIOExtensionPropertyCustomPropertyData_just,
						CMIOExtensionPropertyCustomPropertyData_dust,
						CMIOExtensionPropertyCustomPropertyData_rust,
						nil];
}

- (nullable CMIOExtensionStreamProperties *)streamPropertiesForProperties:(NSSet<CMIOExtensionProperty> *)properties
	error:(NSError * _Nullable *)outError {
NSMutableDictionary* dictionary = [self sampleCustomPropertiesWithPropertySet:properties];
CMIOExtensionStreamProperties* streamProperties = [CMIOExtensionStreamProperties streamPropertiesWithDictionary:dictionary];

This error message below is not related to the actual format of the key, the error is regarding the format of the propertyState (ie. the data associated with the key).

CMIO_DAL_CMIOExtension_Stream.mm:1165:GetPropertyData 50 wrong 4cc format for key 4cc_cust_glob_0000

I haven't tried on the Ventura beta but on 12.4 I am in the practice of rebooting after every update to my extension. If you're having problems it might be due to the fact that an old instance of your extension is being used by the system.

  • Thanks @DrXibber for your insights. Do you think it'd be possible to use one of these custom properties to have the app pass around NSData* representing a frame that ultimately gets shown by the Camera Extension? That way we wouldn't need complicated XPC (which nobody seems to have figured out yet except perhaps @ssmith_c in https://developer.apple.com/forums/thread/706184 ?) or even use another sink stream (also suggested by @ssmith_c).

Add a Comment

Replying to my own question with code to declare custom string and data properties on the camera extension, and getting/setting them on the app side in Swift. https://gist.github.com/ldenoue/84210280853f0490c79473b6edd25e9d