I am trying to build level metering for
AVPlayer
. I am doing this with an
MTAudioProcessingTap
that gets passed to an
AVAudioMix
, which in turns gets passed to the
AVPlayerItem
. The
MTAudioProcessingTap
gets created with the
kMTAudioProcessingTapCreationFlag_PostEffects
flag.
Technical Q&A QA1783 has the following to say about the
PreEffects
and
PostEffects
flags:
When you create a "pre-effects" audio tap using the kMTAudioProcessingTapCreationFlag_PreEffects flag, the tap will be called before any effects specified by AVAudioMixInputParameters are applied; when you create a "post-effects" tap by using the kMTAudioProcessingTapCreationFlag_PostEffects flag, the tap will be called after those effects are applied. Currently the only "effect" that AVAudioMixInputParameters supports is a linear volume ramp.
The problem:
When created with the
kMTAudioProcessingTapCreationFlag_PostEffects
, I would expect the that samples received by the
MTAudioProcessingTap
would reflect the the volume or audio ramps set on the
AVAudioMixInputParameters
. For example, if I set the volume to 0, I would expect to get all 0 samples. However the samples I receive seem to be totally unaffected by the volume or volume ramps.
Am I doing something wrong?
Here is a quick an dirty playground that illustrates the problem. The example sets the volume directly, but I observed the same problem when using audio ramps. Tested on both macOS and iOS:
import Foundation import XCPlayground import PlaygroundSupport import AVFoundation import Accelerate PlaygroundPage.current.needsIndefiniteExecution = true; let assetURL = Bundle.main.url(forResource: "sample", withExtension: "mp3")! let asset = AVAsset(url: assetURL) let playerItem = AVPlayerItem(asset: asset) var audioMix = AVMutableAudioMix() // The volume. Set to > 0 to hear something. let kVolume: Float = 0.0 var parameterArray: [AVAudioMixInputParameters] = [] for assetTrack in asset.tracks(withMediaType: .audio) { let parameters = AVMutableAudioMixInputParameters(track: assetTrack); parameters.setVolume(kVolume, at: kCMTimeZero) parameterArray.append(parameters) // Omitting most callbacks to keep sample short: var callbacks = MTAudioProcessingTapCallbacks( version: kMTAudioProcessingTapCallbacksVersion_0, clientInfo: nil, init: nil, finalize: nil, prepare: nil, unprepare: nil, process: { (tap, numberFrames, flags, bufferListInOut, numberFramesOut, flagsOut) in guard MTAudioProcessingTapGetSourceAudio(tap, numberFrames, bufferListInOut, flagsOut, nil, numberFramesOut) == noErr else { preconditionFailure() } // Assume 32bit float format, native endian: for i in 0..<bufferListInOut.pointee.mNumberBuffers { let buffer = bufferListInOut.pointee.mBuffers let stride: vDSP_Stride = vDSP_Stride(buffer.mNumberChannels) let numElements: vDSP_Length = vDSP_Length(buffer.mDataByteSize / UInt32(MemoryLayout<Float>.stride)) for j in 0..<Int(buffer.mNumberChannels) { // Use vDSP_maxmgv tof ind the maximum amplitude var start = buffer.mData!.bindMemory(to: Float.self, capacity: Int(numElements)) start += Int(j * MemoryLayout<Float>.stride) var magnitude: Float = 0 vDSP_maxmgv(start, stride, &magnitude, numElements - vDSP_Length(j)) DispatchQueue.main.async { print("buff: \(i), chan: \(j), max: \(magnitude)") } } } } ) var tap: Unmanaged<MTAudioProcessingTap>? guard MTAudioProcessingTapCreate(kCFAllocatorDefault, &callbacks, kMTAudioProcessingTapCreationFlag_PostEffects, &tap) == noErr else { preconditionFailure() } parameters.audioTapProcessor = tap?.takeUnretainedValue() } audioMix.inputParameters = parameterArray playerItem.audioMix = audioMix let player = AVPlayer(playerItem: playerItem) player.rate = 1.0