# Analyzing media Use the FxPlug analysis API to analyze frames of source media before rendering them. ## Discussion Frame-by-frame analysis of video clips is commonly required for rendering tasks that occur in an FxPlug plug-in. For example, tracking of objects and stabilization of video both require a pre-render analysis of video frames before rendering. FxPlug 4.1 introduces [`FxAnalysisAPI`](/documentation/professional_video_applications/FxAnalysisAPI), which FxPlug plug-ins can use to analyze video frames from source media in Final Cut Pro X or Motion. `FxAnalysisAPI` uses the host app’s built-in analysis infrastructure so that your plug-in’s analysis can be performed in the background, and you can view the progress UI in the same location as the host’s own analysis processes. This provides a consistent user experience for both native effects and FxPlug plug-ins that require analysis. The example plug-in `FxBrightnessAnalysis` demonstrates how to implement the analysis APIs for a filter that analyzes all frames and then adjusts the brightness of the frames to a consistent level. ### Set the desired analysis time range Your plug-in can tell Motion and Final Cut Pro what time range to analyze. In the following example, using [`desiredAnalysisTimeRange(_:forInputWith:)`](/documentation/professional_video_applications/FxAnalyzer/desiredAnalysisTimeRange(_:forInputWith:)), the `desiredRange` is set to whatever the host provides in `inputTimeRange`. ```objc - (BOOL)desiredAnalysisTimeRange:(CMTimeRange *)desiredRange forInputWithTimeRange:(CMTimeRange)inputTimeRange error:(NSError **)error { // Whatever the host app has, we want to analyze *desiredRange = inputTimeRange; return YES; } ``` ### Set up your plug-in for analysis Set up data arrays or variables that will be used later for analysis in the [`setupAnalysis(for:frameDuration:)`](/documentation/professional_video_applications/FxAnalyzer/setupAnalysis(for:frameDuration:)) method for [`FxAnalyzer`](/documentation/professional_video_applications/FxAnalyzer). You can also use other FxPlug APIs like [`FxParameterSettingAPI_v5`](/documentation/professional_video_applications/FxParameterSettingAPI_v5) within this method to set parameter states during analysis, as in the example below. ```objc - (BOOL)setupAnalysisForTimeRange:(CMTimeRange)analysisRange frameDuration:(CMTime)frameDuration error:(NSError**)error { BOOL result = YES; [analysisLock lock]; { startFrameTime = analysisRange.start; analysisFrameDuration = frameDuration; if (analyzedData != NULL) { free(analyzedData); analyzedData = NULL; } double numFrames = ceil(CMTimeGetSeconds(analysisRange.duration) / CMTimeGetSeconds(frameDuration)); totalFramesToAnalyze = ceil(numFrames); analyzedData = malloc(sizeof(*analyzedData) * (size_t)numFrames); #if BACKWARDS_ANALYSIS nextAnalyzedFrameNum = numFrames - 1; #else nextAnalyzedFrameNum = 0; #endif // BACKWARDS_ANALYSIS } [analysisLock unlock]; return result; } ``` ### Store data to persist between application launches Your plug-in needs to store the analysis data in a parameter so that it can persist between application launches. You typically do this with a custom parameter added in [`addParameters()`](/documentation/professional_video_applications/FxTileableEffect/addParameters()). You also supply the user with a button to start the analysis in [`addParameters()`](/documentation/professional_video_applications/FxTileableEffect/addParameters()). When implementing custom parameters, implement the [`class(forCustomParameterID:)`](/documentation/professional_video_applications/FxTileableEffect/class(forCustomParameterID:)) method so your custom parameters can be serialized and deserialized properly. ```objc - (BOOL)addParametersWithError:(NSError * _Nullable * _Nullable)error { id paramAPI = [_apiManager apiForProtocol:@protocol(FxParameterCreationAPI_v5)]; if (paramAPI == nil) { return NO; } // ... [paramAPI addCustomParameterWithName:@"Analysis Data" parameterID:kParam_AnalysisData defaultValue:[NSDictionary dictionary] parameterFlags:kFxParameterFlag_HIDDEN | kFxParameterFlag_NOT_ANIMATABLE]; return YES; } ``` ```objc - (Class)classForCustomParameterID:(UInt32)parameterID { if (parameterID == kParam_AnalysisData) { return [NSDictionary class]; } return nil; } ``` ### Start the analysis In the `FxBrightnessAnalysis` example plug-in, when the user clicks the Start Analysis push button parameter, the host app calls the plug-in’s `-startAnalysis:` function, which then initializes the [`FxAnalysisAPI`](/documentation/professional_video_applications/FxAnalysisAPI) and requests that the host start a forward analysis.  > Important: > Each plug-in instance is limited to only one analysis job at a time. If you attempt to start another analysis while an analysis is already in progress, the analysis API returns an error. As in the example code below, first check that the ``doc://com.apple.professionalvideoapplications/documentation/professional_video_applications/FxAnalysisAPI/analysisStateForEffect()`` isn’t ``doc://com.apple.professionalvideoapplications/documentation/professional_video_applications/kFxAnalysisState_AnalysisStarted`` or ``doc://com.apple.professionalvideoapplications/documentation/professional_video_applications/kFxAnalysisState_AnalysisRequested`` before attempting to start an analysis. ```objc - (void)startAnalysis:(id)sender { id analysisAPI = [_apiManager apiForProtocol:@protocol(FxAnalysisAPI)]; id colorGamutAPI = [_apiManager apiForProtocol:@protocol(FxColorGamutAPI_v2)]; if ((analysisAPI == nil) || (colorGamutAPI == nil)) { NSLog (@"Unable to retrieve analysis API or color gamut API. Stopping analysis."); return; } FxMatrix44* rgbToYCbCr = [colorGamutAPI colorMatrixFromDesiredRGBToYCbCr]; Matrix44Data* matrixData = [rgbToYCbCr matrix]; luminanceRed = (*matrixData)[0][0]; luminanceGreen = (*matrixData)[0][1]; luminanceBlue = (*matrixData)[0][2]; // If analysis isn't already happening, start it. If it is already happening, just // continue on. FxAnalysisState currentState = [analysisAPI analysisStateForEffect]; if ((currentState != kFxAnalysisState_AnalysisStarted) && (currentState != kFxAnalysisState_AnalysisRequested)) { NSError* err = nil; #if BACKWARDS_ANALYSIS if (![analysisAPI startBackwardAnalysis:kFxAnalysisLocation_CPU error:&err]) #else if (![analysisAPI startForwardAnalysis:kFxAnalysisLocation_CPU error:&err]) #endif // BACKWARDS_ANALYSIS { NSLog (@"Analysis failed to start due to error: %@", err); } } } ``` ### Analyze the frames and store the results Use the [`analyzeFrame(_:at:)`](/documentation/professional_video_applications/FxAnalyzer/analyzeFrame(_:at:)) method allows you to analyze each frame and store results. In the following example the plug-in samples all the pixels to determine the average luminance of each frame, and then stores the `averageLuminance` value for each frame (`frameTime`) in the `analyzedData` array. > Note: > The FxAnalysis API always analyzes all pixels in the frame, instead of just tiles. ```objc - (BOOL)analyzeFrame:(FxImageTile *)frame atTime:(CMTime)frameTime error:(NSError **)error { if (frame.ioSurface.pixelFormat != kCVPixelFormatType_128RGBAFloat) { if (error != nil) { *error = [NSError errorWithDomain:FxPlugErrorDomain code:kFxError_AnalysisError userInfo:@{ NSLocalizedDescriptionKey : @"Invalid bit depth for analysis data" }]; } return NO; } // Calculate the average brightness of the next frames double averageLuminance = 0; NSInteger width = frame.ioSurface.width; NSInteger height = frame.ioSurface.height; [frame.ioSurface lockWithOptions:kIOSurfaceLockReadOnly seed:nil]; NSUInteger rowBytes= frame.ioSurface.bytesPerRow; float* pixels = frame.ioSurface.baseAddress; for (NSInteger row = 0; row < height; row++) { float* nextPixel = (float*)((UInt8*)pixels + (row * rowBytes)); for (NSInteger col = 0; col < width; col++) { double red = *nextPixel; nextPixel++; double green = *nextPixel; nextPixel++; double blue = *nextPixel; nextPixel++; nextPixel++; // skip alpha double luminance = luminanceRed * red + luminanceGreen * green + luminanceBlue * blue; averageLuminance += luminance; } } [frame.ioSurface unlockWithOptions:kIOSurfaceLockReadOnly seed:nil]; averageLuminance /= (double)width * (double)height; [analysisLock lock]; // Add our analysis data to the array NSLog (@"Average luminance for frame at time %f = %f", CMTimeGetSeconds(frameTime), averageLuminance); analyzedData [ nextAnalyzedFrameNum ].frameTime = frameTime; analyzedData [ nextAnalyzedFrameNum ].averageLuminance = averageLuminance; #if BACKWARDS_ANALYSIS nextAnalyzedFrameNum--; #else nextAnalyzedFrameNum++; #endif // BACKWARDS_ANALYSIS // Attempt to save partial data. If it fails, don't worry because we'll save it all at the end // Also, don't save on every frame, as it slows down the analysis too much. Save every 10th // frame instead if ((nextAnalyzedFrameNum % 10) == 0) { if (![self saveAnalyzedData:nil]) { NSLog (@"Unable to save partial analysis data"); } } [analysisLock unlock]; return YES; } ``` The example above calls the `-saveAnalyzedData:` function shown below every 10 frames. The function then saves the results into a custom parameter using the [`setCustomParameterValue(_:toParameter:at:)`](/documentation/professional_video_applications/FxParameterSettingAPI_v5/setCustomParameterValue(_:toParameter:at:)) method of [`FxParameterSettingAPI_v5`](/documentation/professional_video_applications/FxParameterSettingAPI_v5). ```objc - (BOOL)saveAnalyzedData:(NSError**)error { BOOL result = NO; NSData* analysisData = [NSData dataWithBytes:analyzedData length:totalFramesToAnalyze * sizeof (*analyzedData)]; NSDictionary* paramData = [NSDictionary dictionaryWithObjectsAndKeys: (NSDictionary*)CMTimeCopyAsDictionary(analysisFrameDuration, kCFAllocatorDefault), kKey_FrameDuration, analysisData, kKey_AnalysisData, [NSNumber numberWithInteger:nextAnalyzedFrameNum - 1], kKey_NumAnalyzedFrames, nil]; result = YES; id paramAPI = [_apiManager apiForProtocol:@protocol(FxParameterSettingAPI_v5)]; if (paramAPI != nil) { [paramAPI setCustomParameterValue:paramData toParameter:kParam_AnalysisData atTime:kCMTimeZero]; result = YES; } else { if (error != nil) { *error = [NSError errorWithDomain:FxPlugErrorDomain code:kFxError_APIUnavailable userInfo:@{ NSLocalizedDescriptionKey : @"Unable to retrieve the FxParameterSettingAPI_v5 in -cleanupAnalysis:" }]; } } return result; } ``` > Tip: > To cancel analysis, return `NO` and set the error in the ``doc://com.apple.professionalvideoapplications/documentation/professional_video_applications/FxAnalyzer/analyzeFrame(_:at:)`` method. ### Clean up after analysis Use the [`cleanupAnalysis()`](/documentation/professional_video_applications/FxAnalyzer/cleanupAnalysis()) method to do any necessary cleanup after the analysis is complete. The following example is from the `FxBrightnessAnalysis` plug-in. ```objc - (BOOL)cleanupAnalysis:(NSError **)error { // Update our parameters BOOL result = NO; [analysisLock lock]; { [self saveAnalyzedData:error]; free(analyzedData); analyzedData = NULL; result = YES; } [analysisLock unlock]; return result; } ``` ### Determine the Current Analysis State The `-analysisStateForEffect:` method returns the current analysis state as one of the following keys: - [`kFxAnalysisState_NotAnalyzing`](/documentation/professional_video_applications/kFxAnalysisState_NotAnalyzing): Not analyzing. - [`kFxAnalysisState_AnalysisRequested`](/documentation/professional_video_applications/kFxAnalysisState_AnalysisRequested): Analysis has been requested, but it hasn’t started yet. - [`kFxAnalysisState_AnalysisStarted`](/documentation/professional_video_applications/kFxAnalysisState_AnalysisStarted): Analysis has started. - [`kFxAnalysisState_AnalysisCompleted`](/documentation/professional_video_applications/kFxAnalysisState_AnalysisCompleted): Analysis has completed. - [`kFxAnalysisState_AnalysisInterrupted`](/documentation/professional_video_applications/kFxAnalysisState_AnalysisInterrupted): The analysis was interrupted. Although not necessary, you can use these states to provide UI feedback, and to determine when the analysis is complete or whether it was interrupted. ### Retrieve analysis data for use in rendering During rendering, retrieve the analyzed data for each frame from the custom parameter in your [`pluginState(_:at:quality:)`](/documentation/professional_video_applications/FxTileableEffect/pluginState(_:at:quality:)) method. Use [`getCustomParameterValue(_:fromParameter:at:)`](/documentation/professional_video_applications/FxParameterRetrievalAPI_v6/getCustomParameterValue(_:fromParameter:at:)) to get the frame’s value, process a result that your renderer will need, then pack it into the plug-in state for that frame. This example shows how to retrieve the analysis data for use in rendering. The steps that follow take you through the example in more detail. ```objc - (BOOL)pluginState:(NSData * _Nonnull * _Nullable)pluginState atTime:(CMTime)renderTime quality:(FxQuality)qualityLevel error:(NSError * _Nullable * _Nullable)error { id paramAPI = [_apiManager apiForProtocol:@protocol(FxParameterRetrievalAPI_v6)]; if (paramAPI == nil) { return NO; } NSDictionary* analysisData = nil; [paramAPI getCustomParameterValue:&analysisData fromParameter:kParam_AnalysisData atTime:renderTime]; BOOL completedAnalysis = [[analysisData allKeys] count] > 0; CMTime frameDuration = kCMTimeInvalid; if (completedAnalysis) { frameDuration = CMTimeMakeFromDictionary((CFDictionaryRef)[analysisData objectForKey:kKey_FrameDuration]); } NSInteger numFramesAnalyzed = [[analysisData objectForKey:kKey_NumAnalyzedFrames] integerValue]; NSData* analyzedParamData = [analysisData objectForKey:kKey_AnalysisData]; BrightnessAnalysis* brightnessAnalysis = (BrightnessAnalysis*)[analyzedParamData bytes]; double actualBrightness = 1.0; if (completedAnalysis) { actualBrightness = [self brightnessAtTime:renderTime frameDuration:frameDuration numFrames:numFramesAnalyzed fromData:brightnessAnalysis]; } double desiredBrightness = 1.0; [paramAPI getFloatValue:&desiredBrightness fromParameter:kParam_Brightness atTime:renderTime]; id colorAPI = [_apiManager apiForProtocol:@protocol(FxColorGamutAPI_v2)]; FxMatrix44* rgbToYCbCr = [colorAPI colorMatrixFromDesiredRGBToYCbCr]; FxMatrix44* YCbCrToRGB = [colorAPI colorMatrixFromYCbCrToDesiredRGB]; Matrix44Data *rgbToYCbCrData = [rgbToYCbCr matrix]; Matrix44Data *YCbCrToRGBData = [YCbCrToRGB matrix]; struct BrightnessEqualization brightnessData = { completedAnalysis, desiredBrightness, actualBrightness, frameDuration, [self matrixDataToFlatArray:rgbToYCbCrData], [self matrixDataToFlatArray:YCbCrToRGBData] }; NSMutableData* renderData = [NSMutableData dataWithBytes:&brightnessData length:sizeof(brightnessData)]; *pluginState = renderData; return YES; } ``` In the above example from the `FxBrightnessAnalysis` plug-in: 1. The `analysisData` for the current frame is retrieved from the custom parameter using [`getCustomParameterValue(_:fromParameter:at:)`](/documentation/professional_video_applications/FxParameterRetrievalAPI_v6/getCustomParameterValue(_:fromParameter:at:)). 2. A pointer to the data is then passed to a function called `brightnessAtTime::::` to determine the analyzed brightness value of the current frame. 3. The `desiredBrightness` value that was set in the `kParam_Brightness` float slider parameter by the user is retrieved. 4. The `actualBrightness` and `desiredBrightness` values are packed into the `brightnessData` struct and returned as `renderData`. See [Communicating with the plug-in state](/documentation/Professional-Video-Applications/communicating-with-the-plug-in-state) for more information on working with `pluginState`. --- Copyright © 2026 Apple Inc. All rights reserved. | [Terms of Use](https://www.apple.com/legal/internet-services/terms/site.html) | [Privacy Policy](https://www.apple.com/privacy/privacy-policy)