Dive into the technical aspects of audio on your device, including codecs, format support, and customization options.

Audio Documentation

Posts under Audio subtopic

Post

Replies

Boosts

Views

Activity

Is AVMutableVideoComposition missing the customVideoCompositor property?
Hi,According to the documentation for the AVVideoCompositing protocol:"When creating instances of custom video compositors, AV Foundation initializes them by calling init and then makes them available as the value of the customVideoCompositor property of the object to which it was assigned. You then can do any additional setup or configuration to the custom compositor."AVMutableVideoComposition has a customVideoCompositorClass property, but does not have a customVideoCompositor property.Am I misunderstanding this? I need to access the instance to set some properties on it, but I cannot.Thanks,Frank
1
0
1.1k
Apr ’22
App Review - Background Audio capabilities not accepted
Hi,I have asubmitted an application to teh App Review Board. They have rejected the app with the following reason:"Your app declares support for audio in the UIBackgroundModes key in your Info.plist but did not include features that require persistent audio."My app uses audio recording in the background and asks the user for microphone permissions when they first use the application. I have also appealed the decision but they say that the way we are using audio in the background is not acceptable and still rejected the app. I am happy to make any changes that may be needed but I want clarification on what the problem is because according to the guidlines for background excecution, what I am doing is acceptable. Does anyone know how I can talk to an Apple engineer for her/him to explain to me what needs to be done? I am stuck and we have spent over a year developing this app. Best,Feras A.
2
0
4.7k
May ’22
macOS console application with AVCaptureDevice
I am working on a console app (Actually a library, but designed to be used from console), where I want to be able to enumerate the AVCaptureDevices at runtime. Attempting to do this however, I am noticing that the AVCaptureDevice devices property is never updating. I have some minimal code to reproduce this. Run it with a USB camera plugged in, and as you unplug and replug the camera in the output never changes, and it says there are devices that don't actually exist. Note I am not running this in a UI application, so there is no standard loop. Its just meant to be running from a standard "c" main.#import <Foundation/Foundation.h> #import <AVFoundation/AVFoundation.h> int main(int argc, const char * argv[]) { @autoreleasepool { while (true) { NSArray* captureDevices = [AVCaptureDevice devicesWithMediaType:AVMediaTypeVideo]; for (id device in captureDevices) { NSString* uniqueIdentifier = [(AVCaptureDevice*)device uniqueID]; NSLog(uniqueIdentifier); } usleep(1000000); } } return 0; }Thanks for any help. I would be ok firing up a message loop in a background thread if I needed to, but I can't guarentee that the user application has a message loop (sorry if terminology is different, I come from windows).
4
0
1.1k
Dec ’21
aumf in logic pro x (10.4)
Hi, I've modified the AU Filter Demo to accept MIDI notes to control a parameter. I changed the type to aumf. It validates and loads into logic pro x 10.4 just fine.However, when I create a software instrument and go to insert my midi controlled effect in the input slot, it doesn't show up. I also notice that there aren't any other midi controlled effects in the plugin manager, so I don't have a reference case.Am I instantiating the audio unit correctly in logic pro (I couldn't find anything in the help docs for logic)? Is there something else I need to configure or set in the audio unit so that logic "sees" it correctly?Thanks!Chris
6
0
3.8k
Mar ’23
When playing video Content : KVO_IS_RETAINING_ALL_OBSERVERS_OF_THIS_OBJECTS_IF_IT_CRASHES_AN_OBSERVER_WAS_OVERRELEASED_OR_SMASHED
When i am playing video got this crash.Crashlytics log reportsCrashed: com.apple.main-thread0 libobjc.A.dylib 0x1845a67e8 object_isClass + 161 Foundation 0x185d103e8 KVO_IS_RETAINING_ALL_OBSERVERS_OF_THIS_OBJECT_IF_IT_CRASHES_AN_OBSERVER_WAS_OVERRELEASED_OR_SMASHED + 682 Foundation 0x185d0e8ec NSKeyValueWillChangeWithPerThreadPendingNotifications + 3003 AVFoundation 0x18abb60f8 -[AVPlayerItem willChangeValueForKey:] + 964 AVFoundation 0x18abc7ff0 -[AVPlayerItem _updatePropertyCacheAndTriggerKVOForPlayer:usingKeys:] + 2005 AVFoundation 0x18abb959c __60-[AVPlayerItem _informObserversAboutAvailabilityOfDuration:]_block_invoke_2 + 1846 libdispatch.dylib 0x184cdd088 _dispatch_call_block_and_release + 247 libdispatch.dylib 0x184cdd048 _dispatch_client_callout + 168 libdispatch.dylib 0x184d1ddfc _dispatch_main_queue_callback_4CF$VARIANT$armv81 + 9689 CoreFoundation 0x185301eb0 __CFRUNLOOP_IS_SERVICING_THE_MAIN_DISPATCH_QUEUE__ + 1210 CoreFoundation 0x1852ffa8c __CFRunLoopRun + 201211 CoreFoundation 0x18521ffb8 CFRunLoopRunSpecific + 43612 GraphicsServices 0x1870b7f84 GSEventRunModal + 10013 UIKit 0x18e7f42f4 UIApplicationMain + 20814 VZFiOSMobile 0x10475b340 main (main.m:17)15 libdyld.dylib 0x184d4256c start + 4Why does this happens and what I can do to avoid this?
9
0
8.8k
Nov ’21
(iOS) Issues with 'aumi' Audio Units and AVAudioEngine
Hi all,I just filled the bug report 39600781, regarding 'aumi' (kAudioUnitType_MIDIProcessor) audio units and AVAudioEngine. In short: in iOS, when attaching 'aumi' audio units to an AVAudioEngine, the AVAudioEngine will never call their renderBlock / internalRenderBlock, rendering them (pun 😝) useless.I write this post because I know that 'aumi' audio units in iOS are undocumented, thus some quirks are to be expected 🙂. But since some developers are already implementing them (I am beta testing their apps), I think it is important to raise awareness that 'aumi' audio units do not work with hosts that use AVAudioEngine. Note that hosts using AUGraph work, because they need to explicitly call the renderBlock method of their audio units. Yet since AUGraph is going to be deprecated... well, if "aumis" are to be officially introduced in the next iOS version, it is important to raise awareness about this particular issue. That's all. Thanks for reading 😉.
7
0
2.7k
Oct ’22
Switch camera while recording video?
Hi,I have an app that includes a very simple one-button interface for recording a video. Our customer wants to be able to switch between the front and rear cameras while the video is being recorded, and without any interruption in the video stream. I notice that even the iOS built-in camera app doesn't do this, but I've heard that some third-party apps do.Is this possible in a practical way? If so, how would I go about it?Thanks,Frank
3
0
15k
Nov ’21
Another photo orientation question
HiI'm using Swift 4 and the AvCam example code from Apple. In my own app I've an issue where the captured image orientation is wrong. My requirement is to save the photos to my applications folder as opposed to the Photos library. I've spent a day trying to correct my own code and got nowhere.As a test I modified the AvCam code to write the photo data to a static variable in a struct and then display that image on the camera view in AvCam by tapping a button as follows:// In the camera view controller: @IBOutlet weak var pv: UIImageView! // for previewing test @IBAction func btn(_ sender: UIButton) { // for setting preview test image let dataProvider = CGDataProvider(data: photoTest.photo! as CFData) let cgImageRef: CGImage! = CGImage(jpegDataProviderSource: dataProvider!, decode: nil, shouldInterpolate: true, intent: .defaultIntent) let photo = UIImage(cgImage: cgImageRef) pv.contentMode = .scaleAspectFill pv.image = photo }The orientaion issue is replicated in AvCam with the above code. In my own app the image is previewed before saving to the apps folder and the images orientation is always wrong in the preview as well as when reloading the image from disk. I.e. if the iphone is in portrait orientation, the preview image is rotated 90 degrees anti-clockwise, in landscape the same happens so the preview image appears to be the correct way up.I'm assuming I've completely missed the point somewhere along the line as it seems to me that regardless of device orientation, the top of a photo should always be the top. For example, in the Photos app photos are shown the correct way up regardless of orientation.Any help would be much appreciated, this is driving me nuts ;o)Cheers
5
0
4.5k
Feb ’22
Remote commands not delivered if AVPlayer is paused
I have a simple tvOS app that has a single audio track, a storyboard with single scene that contains a button for play/pause toggling.I am trying to add Siri Remote handling so that user can play/pause the track with remote's "toggle play/pause" button.The problem, though, is that when the application starts it doesn't respond to remote control events if the AVPlayer is paused. When user starts playback using the touch-button on Siri Remote I can pause the playback using "toggle play/pause" button. I CAN'T start playback again after the track has been paused.I tried two approaches:application.beginReceivingRemoteControlEvents()MPRemoteCommandCenter.shared().pauseCommand.addTarget(...)Using the remoteControlReceived(with event: UIEvent?) I receive the event only when the AVPlayer is playing and the event's subtype is pause command.Using the remote command center I registered for pauseCommand, playCommand and togglePlayPauseCommandcommands but only pauseCommand is being triggered when the playback is active.Am I missing something?
1
0
2.5k
Dec ’21
Implementing camera switch (front to back, vice versa)
I'm having trouble understand how to switch camera from front to back, back to front upon the touch of a button.My current code doesn't work.I get " Multiple audio/video AVCaptureInputs are not currently supported'Here is what I have so far:import UIKit import AVFoundation import Photos class CameraViewController :UIViewController { func applyRoundCorner(_ object: AnyObject){ object.layer.cornerRadius = (object.frame.size.width)/2 object.layer.masksToBounds = true } @IBOutlet weak var cameraButton: UIButton! @IBOutlet weak var imagePreview: UIImageView! @IBAction func Library(_ sender: Any) { } @IBAction func FlipCamera(_ sender: Any) { captureSession.stopRunning() cameraPreviewlayer?.removeFromSuperlayer() //cameraPreviewlayer = nil //self.captureSession = nil setupCaptureSession() if currentCamera! == backCamera{ //print(currentCamera) currentCamera = frontCamera} else { currentCamera = backCamera} setupCaptureSession() setupInputOutput() setupPreviewLayer() startRunningCaptureSession() } var captureSession = AVCaptureSession() var backCamera: AVCaptureDevice? var frontCamera: AVCaptureDevice? var currentCamera: AVCaptureDevice? var cameraPreviewlayer: AVCaptureVideoPreviewLayer? var photoOutput: AVCapturePhotoOutput? var image: UIImage? override func viewDidLoad() { super.viewDidLoad() self.applyRoundCorner(cameraButton) setupCaptureSession() setupInputOutput() setupPreviewLayer() startRunningCaptureSession() } func setupCaptureSession(){ captureSession.sessionPreset = AVCaptureSession.Preset.photo // Why exclamation point? let availableDevice = AVCaptureDevice.DiscoverySession(deviceTypes: [AVCaptureDevice.DeviceType.builtInWideAngleCamera], mediaType: .video, position: .unspecified) let devices = availableDevice.devices //back or front for device in devices { if device.position == AVCaptureDevice.Position.back{ backCamera = device }else if device.position == AVCaptureDevice.Position.front{ frontCamera = device } } currentCamera = backCamera } func setupInputOutput(){ do { let captureDeviceInput = try AVCaptureDeviceInput(device: currentCamera!) captureSession.addInput(captureDeviceInput) photoOutput = AVCapturePhotoOutput() photoOutput?.setPreparedPhotoSettingsArray([AVCapturePhotoSettings(format:[AVVideoCodecKey:AVVideoCodecType.jpeg])], completionHandler: nil) captureSession.addOutput(photoOutput!) } catch{ print(error) } } func setupPreviewLayer(){ cameraPreviewlayer = AVCaptureVideoPreviewLayer(session: captureSession) cameraPreviewlayer?.videoGravity = AVLayerVideoGravity.resizeAspectFill cameraPreviewlayer?.connection?.videoOrientation = AVCaptureVideoOrientation.portrait cameraPreviewlayer?.frame = self.imagePreview.frame self.view.layer.insertSublayer(cameraPreviewlayer!, at: 0) } func startRunningCaptureSession(){ captureSession.startRunning() } @IBAction func TakePhoto(_ sender: Any) { let settings = AVCapturePhotoSettings() photoOutput?.capturePhoto(with: settings, delegate: self) } override func prepare(for segue: UIStoryboardSegue, sender: Any?){ if segue.identifier == "showPhotoSegue"{ let previewVC = segue.destination as! PreviewViewController previewVC.image = self.image } } } extension CameraViewController: AVCapturePhotoCaptureDelegate{ func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto photo: AVCapturePhoto, error: Error?) { if let imageData = photo.fileDataRepresentation(){ image = UIImage(data: imageData) performSegue(withIdentifier:"showPhotoSegue", sender: nil) } } }
6
0
4k
Jan ’23
Audio renderer fails to render CMSampleBuffer
I am trying to render audio using AVSampleBufferAudioRenderer but there is no sound coming from my speakers and there is a repeated log message.[AQ] 405: SSP::Render: CopySlice returned 1I am creating a CMSampleBuffer from an AudioBufferList. This is the relevant code:var sampleBuffer: CMSampleBuffer! try runDarwin(CMSampleBufferCreate(allocator: kCFAllocatorDefault, dataBuffer: nil, dataReady: false, makeDataReadyCallback: nil, refcon: nil, formatDescription: formatDescription, sampleCount: sampleCount, sampleTimingEntryCount: 1, sampleTimingArray: &timingInfo, sampleSizeEntryCount: sampleSizeEntryCount, sampleSizeArray: sampleSizeArray, sampleBufferOut: &sampleBuffer)) try runDarwin(CMSampleBufferSetDataBufferFromAudioBufferList(sampleBuffer, blockBufferAllocator: kCFAllocatorDefault, blockBufferMemoryAllocator: kCFAllocatorDefault, flags: 0, bufferList: audioBufferList.unsafePointer)) try runDarwin(CMSampleBufferSetDataReady(sampleBuffer))I am pretty confident that my audio format description is correct because CMSampleBufferSetDataBufferFromAudioBufferList, which performs a laundry list of validations, returns no error.I tried to reverse-engineer the CopySlice function, but I’m lost without the parameter names.int ScheduledSlicePlayer::CopySlice( long long, ScheduledSlicePlayer::XScheduledAudioSlice*, int, AudioBufferList&, int, int, bool )Does anyone have any ideas on what’s wrong? For the Apple engineers reading this, can you tell me the parameter names of the CopySlice function so that I can more easily reverse-engineer the function to see what the problem is?
4
1
3.3k
Oct ’22
AudioComponentInstanceNew takes several seconds to complete
I'm using a VoiceProcessingIO audio unit in my VoIP application on Mac. The problem is, at least since Mojave, AudioComponentInstanceNew blocks for at least 2 seconds. Profiling shows that internally it's waiting on some mutex and then on some message queue. My code to initialize the audio unit is as follows: OSStatus status; AudioComponentDescription desc; AudioComponent inputComponent; desc.componentType = kAudioUnitType_Output; desc.componentSubType = kAudioUnitSubType_VoiceProcessingIO; desc.componentFlags = 0; desc.componentFlagsMask = 0; desc.componentManufacturer = kAudioUnitManufacturer_Apple; inputComponent = AudioComponentFindNext(NULL, &desc); status = AudioComponentInstanceNew(inputComponent, &unit);Here's a profiler screenshot showing the two system calls in question.So, is this a bug or an intended behavior?
4
0
2.6k
Jul ’23
double output of a camera video stream
Hello, everybody,I would like to display the camera image of the back camera twice on the screen (as with the cardboard). I created two views and they each take 50% of the screen, I can also output the live image of the camera on one view.// views left and right var viewRect:CGRect! var leftView:UIView! var rightView:UIView! //... //left viewRect = CGRect(x: 0, y: 0, width: 0.5 * self.view.frame.width, height: self.view.frame.height) leftView = UIView(frame: viewRect) leftView.backgroundColor = .green view.addSubview(leftView) //rgiht viewRect = CGRect(x: 0.5 * self.view.frame.width, y: 0, width: 0.5 * self.view.frame.width, height: self.view.frame.height) rightView = UIView(frame: viewRect) rightView.backgroundColor = .red view.addSubview(rightView) //... func beginSession(){ captureSession.sessionPreset = AVCaptureSession.Preset.photo let devices = AVCaptureDevice.devices() // Loop through all the capture devices on this phone for device in devices { // Make sure this particular device supports video if (device.hasMediaType(AVMediaType.video)) { // Finally check the position and confirm we've got the back camera if(device.position == AVCaptureDevice.Position.back) { captureDevice = device as? AVCaptureDevice } } } do { try captureSession.addInput(AVCaptureDeviceInput(device: captureDevice!)) captureSession.sessionPreset = AVCaptureSession.Preset.photo } catch _ { } let previewLayer = AVCaptureVideoPreviewLayer(session: captureSession) previewLayer.connection?.videoOrientation = AVCaptureVideoOrientation.landscapeLeft previewLayer.videoGravity = AVLayerVideoGravity.resizeAspectFill self.leftPreviewLayer = previewLayer self.leftPreviewLayer.frame = self.view.layer.frame self.leftPreviewLayer.frame = self.leftView.bounds self.leftView.layer.addSublayer(previewLayer) self.leftView.layer.addSublayer(previewLayer) captureSession.startRunning() }I already found something like "let replicatorLayer = CAReplicatorLayer()" online, but can't get it to work 😟it would be great if someone could give me a hint 😉Thx and cya Tom
1
0
1.2k
Apr ’22
When will AVReaderWriter example be updated?
The AVReaderWriter was last updated in 2016 for Swift 2.3. I am a newby, and have been unable to upgrade or convert it using the Xcode tools. I tried going from Swift 2 to 3, in preparation of going from 3 to 4, but got a ton of errors. I downloaded every version of Xcode since 8.0 to try and incrementally upgrade it, and it quickly fell apart. Is there an update schedule for their documentation? I hear Swift 5 is coming out, and I am sure 6 and 7 are not far behind. I'm not sure how people are supposed to figure this stuff out if the documentation is several versions out-of-date, I guess it is easier if you have a lot of experience.
11
0
3.1k
Dec ’21
AVAudioEngine returning silence at the beginning of manual rendering
I am using AVAudioEngine to analyse some audio tracks and quite often the `while engine.manualRenderingSampleTime < sourceFile.length {` loop takes a moment to start receiving audio data. Looking at the soundwave looks like the input is simply delayed. This wouldn't be a problem if the length of the buffer would vary depending on this latency but the length unfortunately stays always the same, loosing the final part of the track.I took the code from the official tutorial and this seems to happen regardless if I add an EQ effect or not. Actually looks like the two analysis (with or without EQ) done one after the other return the same anomaly. let format: AVAudioFormat = sourceFile.processingFormat let engine = AVAudioEngine() let player = AVAudioPlayerNode() engine.attach(player) if compress { let eq = AVAudioUnitEQ(numberOfBands: 2) engine.attach(eq) let lowPass = eq.bands[0] lowPass.filterType = .lowPass lowPass.frequency = 150.0 lowPass.bypass = false let highPass = eq.bands[1] highPass.filterType = .highPass highPass.frequency = 100.0 highPass.bypass = false engine.connect(player, to: eq, format: format) engine.connect(eq, to: engine.mainMixerNode, format: format) }else{ engine.connect(player, to: engine.mainMixerNode, format: format) } do { let maxNumberOfFrames: AVAudioFrameCount = 4096 try engine.enableManualRenderingMode(.offline, format: format, maximumFrameCount: maxNumberOfFrames) } catch { fatalError("Could not enable manual rendering mode, \(error)") } player.scheduleFile(sourceFile, at: nil) do { try engine.start() player.play() }catch{ fatalError("Could not start engine, \(error)") } // buffer to which the engine will render the processed data let buffer: AVAudioPCMBuffer = AVAudioPCMBuffer(pcmFormat: engine.manualRenderingFormat, frameCapacity: engine.manualRenderingMaximumFrameCount)! // var pi = 0 // while engine.manualRenderingSampleTime < sourceFile.length { do { let framesToRender = min(buffer.frameCapacity, AVAudioFrameCount(sourceFile.length - engine.manualRenderingSampleTime)) let status = try engine.renderOffline(framesToRender, to: buffer) switch status { case .success: // data rendered successfully let flength = Int(buffer.frameLength) points.reserveCapacity(pi + flength) if let chans = buffer.floatChannelData?.pointee { let left = chans.advanced(by: 0) let right = chans.advanced(by: 1) for b in 0..<flength { let v:Float = max(abs(left[b]), abs(right[b])) points.append(v) } } pi += flength case .insufficientDataFromInputNode: // applicable only if using the input node as one of the sources break case .cannotDoInCurrentContext: // engine could not render in the current render call, retry in next iteration break case .error: // error occurred while rendering fatalError("render failed") } } catch { fatalError("render failed, \(error)") } } player.stop() engine.stop()I thought it was perhaps a simulator issue, but also on the device is happening. Am I doing anything wrong? Thanks!
2
0
2.9k
Oct ’21
When Launching Camera App Crashes
I am trying to access the camera to take pictures in my app. Here is my code:@IBAction func uploadPicture(_ sender: Any) { imagePickerController = UIImagePickerController() imagePickerController.delegate = self imagePickerController.sourceType = .camera present(imagePickerController, animated: true, completion: nil) } func saveImage(imageName: String){ //create an instance of the FileManager let fileManager = FileManager.default //get the image path let imagePath = (NSSearchPathForDirectoriesInDomains(.documentDirectory, .userDomainMask, true)[0] as NSString).appendingPathComponent(imageName) //get the image we took with camera if pic4 != nil { let image = pic4 //get the PNG data for this image let data = image.jpegData(compressionQuality: 1) //store it in the document directory fileManager.createFile(atPath: imagePath as String, contents: data, attributes: nil) // print("Big Bobs\(image.imageOrientation.rawValue)") if image.size.height > image.size.width { newWine.isPortrait = true } print("Bigmacs \(image.imageOrientation.rawValue)") } } func imagePickerController(_ picker: UIImagePickerController, didFinishPickingMediaWithInfo info: [UIImagePickerController.InfoKey : Any]) { //// Local variable inserted by Swift 4.2 migrator. //let info = convertFromUIImagePickerControllerInfoKeyDictionary(info) // // // if let image2 = info[convertFromUIImagePickerControllerInfoKey(UIImagePickerController.InfoKey.originalImage)] as? UIImage { // // newWine.picture = image2 // // // } // // self.dismiss(animated: true, completion: nil) imagePickerController.dismiss(animated: true, completion: nil) // makes image show in image view if I want pic4 = (info[UIImagePickerController.InfoKey.originalImage] as? UIImage)! // imageView.image = info[UIImagePickerController.InfoKey.originalImage] as? UIImage SVProgressHUD.showSuccess(withStatus: "Picture Added Successfully") newWine.pictureIncluded = true }However when I press the upLoadPicture button the app crashes. Everything works fine when using the .photoLibrary in line 5.My info.plist includes the following:Privacy - Photo Library Additions Usage DescriptionPrivacy - Photo Library Usage DescriptionPrivacy  -  Camera Usage DescriptionThanks to all help!
5
0
4.1k
Aug ’22
How to specify bit rate when writing with AVAudioFile
I can currently write, using AVAudioFile, to any of the file formats specified by Core Audio.It can create files in all formats (except one, see below ) that can be read into iTunes, Quicktime and other apps and played back.However some formats appear to be ignoring values in the AVAudioFile settings dictionary.e.g: • An MP4 or AAC will save and write successfully at any sample rate but any bit rates I attempt to specify are ignored. • Wave files saved with floating point data are always converted to Int32 even though I specify float. Even though the PCM buffers I’m using as input and output for sample rate conversion are float on input and output. So the AVAudioFile is taking Float input but converting it to Int for some reason I can’t fathom. • The only crash/exception/failure I see is if I attempt to create an AVAudioFile as WAV/64 bit float. … bang, AVAudioFile isn’t having that one!The technique I’m using is: • Create AVAudioFile for writing with a settings dictionary. • Get processing and file format from AVAudioFile • Client format is always 32 bit Float, AVAudioFile generally reports its processing format as some other word sized Float format at the sample rate and size I’ve specified in the fileFormat. • Create a converter to convert from client format to processing format. • Process input data through the converter to the file using converter.convert(to: , error:&error, withInputFrom )So this works … sort ofThe files ( be they wav, aiff, flac, pp3, aac, mp4 etc ) are written out and will play back just fine.… but …If the processing word format is Float, in a PCM file like WAV, the AVAudioFile will always report its fileFormat as Int32.And if the file is a compressed format such as mp4/aac, any bit rates I attempt to specify are just ignored but the sample rate appears to be respected as if the converters/encoders just choose a bit rates based on sample rate.So after all that waffle, I've missed something that's probably meant to be obvious, so my questions are … • For lpcm float formats why is Int32 data written even though the AVAudioFile settings dictionary has AVLinearPCMIsFloatKey to true ? • How do arrange the setup so that I can specify the bit rate for compressed audio?The only buffers I actually create are both PCM, the client output buffer, and the AVAudioConverter/AVAudioFile processing buffer.I’ve attempted using AVAudioCompressedBuffer but haven’t had any luck.I hope someone has some clues because I’ve spent more hours on this than anyone should ever need to!For my Christmas present I’d like Core Audio to be fully and comprehensively documented please!
4
0
4.9k
Aug ’21
Why does Apple use simplified 1.961 gamma instead of precise ITU-R 709 transfer function?
(For Apple folks: rdar://47577096.)# BackgroundThe Core Video function `CVImageBufferCreateColorSpaceFromAttachments` creates custom color profiles with simplified transfer functions instead of using the standard system color profiles. Let’s take ITU-R 709 as an example.The macOS `Rec. ITU-R BT.709-5` system color profile specifies the transfer function asf(x) = { (0.91x + 0.09)^2.222 where x >= 0.081 { 0.222x where x < 0.081The Apple-custom `HDTV` color profile created by the above Core Video function specifies the transfer function asf(x) = x^1.961My understanding is that `x^1.961` is the closest approximation of the more complex ITU-R 709 transfer function.# Questions1. Why use a custom color profile with a simplified transfer function rather than the official specification? - Was it done for performance? - Was it done for compatibility with non-QuickTime-based applications? - etc.2. Speaking of compatibility, there is a problem when an encoding application uses the official transfer function and the decoding application uses the approximated transfer function. I tested this using two images. One image uses the `Rec. ITU-R BT.709-5` color profile. The other image is derived from the former by assigning the Apple-custom `HDTV` color profile. The latter image loses the details in the darker areas of the image. Why go to the trouble of approximating the transfer function when the approximation isn’t that great?3. Are the Apple-custom color profiles also used for encoding? Or are they only for decoding?4. Another thing that concerns me is that the Apple-custom `HDR (PQ)` and `HDR (HLG)` color profiles use the same simplified transfer function of `f(x) = x^1.801`. Isn’t the whole point of the PQ and HLG standards to define more sophisticated transfer functions? Doesn’t simplifying those two transfer functions defeat their purpose?
1
1
3.8k
Apr ’22
Trouble getting good bluetooth audio behavior
We are trying to get good bluetooth behavior for our app but there are a whole slew of problems.There are 2 issues that I don't understand how to resolve,1. I don't receive notifications from AVAudioSessionRouteChangeNotification if user starts app without Bluetooth headphones connected.2. If I stream with Apple Music, after adding the code to enable Bluetooth, all audio from my app is lost unless I put app in background then bring it to the foreground at which the audio is now forced into the earpiece speaker phoneWithout making any changes to the code to try and incorporate bluetooth capabilities our app currently forces audio through the speakers.To enable bluetooth we've added the following code just before the app finishes initializing which works great if the user already has their bluetooth headphones connected. The audio will successfully be outputed into the headphones. If they disconnect their headphones the audio will be outputted to the device speakers.[[AVAudioSession sharedInstance] setCategory:AVAudioSessionCategoryPlayAndRecord withOptions:AVAudioSessionCategoryOptionDefaultToSpeaker | AVAudioSessionCategoryOptionAllowBluetoothA2DP | AVAudioSessionCategoryOptionAllowAirPlay error:nil]; [[AVAudioSession sharedInstance] setActive:YES error:nil];However if the user is streaming music through our app then behavior becomes very poor. We have 2 ways of streaming music, the first is using an AVPlayer to stream music from a URL, the other way is using MPMusicPlayerController to stream music from Apple Music.After adding the code to enable bluetooth wireless headphones, if the user starts streaming music with Apple Music, all audio such as sound effects from our app get turned off and the only thing the user can hear is the music being streamed. If they stop streaming the music the audio never comes back. If they then turn off the bluetooth wireless headphones and put the app into the background followed by bringing it back into the foreground, the sound effects come back but now they are outputted through the earphone speaker.That is the worst of all the behavior we see.When streaming with the AVPlayer we don't see any issues with the app sound effects being lost but, if the user disconnects their bluetooth wireless headphones while streaming I cannot get the app to detect when they become available again if the user turns them on. I have the following code to listen when the audio routes change and that works great when disconnecting the bluetooth wireless headphones, but when reconnecting them I can't get a notification unless after I've turned my bluetooth headphones on that I put the app into the background and bring it back into the foreground.NSNotificationCenter* notificationCenter = [NSNotificationCenter defaultCenter]; [notificationCenter addObserver:media_sharedInstance selector:@selector(audioHardwareRouteChanged:) name:AVAudioSessionRouteChangeNotification object:nil];- (void) audioHardwareRouteChanged:(NSNotification *)notification{ NSInteger routeChangeReason = [notification.userInfo[AVAudioSessionRouteChangeReasonKey] integerValue]; NSLog(@"route change"); NSLog(@"%ld",(long)routeChangeReason); NSLog(@"output volume %f",[[AVAudioSession sharedInstance] outputVolume]); if(routeChangeReason == AVAudioSessionRouteChangeReasonOldDeviceUnavailable) { //[media_sharedInstance RemoveAudioChange]; NSLog(@"old device unavailable"); [media_sharedInstance ConfigureSpeakers]; //[media_sharedInstance SubscribeToAudioChange]; } if(routeChangeReason == kAudioSessionRouteChangeReason_NewDeviceAvailable) { NSLog(@"new device available"); //[media_sharedInstance RemoveAudioChange]; [media_sharedInstance ConfigureSpeakers]; //[media_sharedInstance SubscribeToAudioChange]; } } - (void) ConfigureSpeakers { [[AVAudioSession sharedInstance] setCategory:AVAudioSessionCategoryPlayAndRecord withOptions:AVAudioSessionCategoryOptionDefaultToSpeaker | AVAudioSessionCategoryOptionAllowBluetooth | AVAudioSessionCategoryOptionAllowAirPlay error:nil]; [[AVAudioSession sharedInstance] setActive:YES error:nil]; }Edit* I figured out the issue with AVAudioSessionRouteChangeNotification not sending a notification. The solution was to use AVAudioSessionCategoryOptionAllowBluetoothA2DP and AVAudioSessionCategoryOptionAllowBluetooth when I received the notification that the bluetooth was disconnected, and when I get the notification that bluetooth becomes available I switch the options to just AVAudioSessionCategoryOptionAllowBluetoothA2DPStill trying to figure out why all of the audio from the app gets shut down when playing Apple Music (MPMusicPlayerController)
1
0
2.3k
Sep ’22
Is AVMutableVideoComposition missing the customVideoCompositor property?
Hi,According to the documentation for the AVVideoCompositing protocol:"When creating instances of custom video compositors, AV Foundation initializes them by calling init and then makes them available as the value of the customVideoCompositor property of the object to which it was assigned. You then can do any additional setup or configuration to the custom compositor."AVMutableVideoComposition has a customVideoCompositorClass property, but does not have a customVideoCompositor property.Am I misunderstanding this? I need to access the instance to set some properties on it, but I cannot.Thanks,Frank
Replies
1
Boosts
0
Views
1.1k
Activity
Apr ’22
App Review - Background Audio capabilities not accepted
Hi,I have asubmitted an application to teh App Review Board. They have rejected the app with the following reason:"Your app declares support for audio in the UIBackgroundModes key in your Info.plist but did not include features that require persistent audio."My app uses audio recording in the background and asks the user for microphone permissions when they first use the application. I have also appealed the decision but they say that the way we are using audio in the background is not acceptable and still rejected the app. I am happy to make any changes that may be needed but I want clarification on what the problem is because according to the guidlines for background excecution, what I am doing is acceptable. Does anyone know how I can talk to an Apple engineer for her/him to explain to me what needs to be done? I am stuck and we have spent over a year developing this app. Best,Feras A.
Replies
2
Boosts
0
Views
4.7k
Activity
May ’22
macOS console application with AVCaptureDevice
I am working on a console app (Actually a library, but designed to be used from console), where I want to be able to enumerate the AVCaptureDevices at runtime. Attempting to do this however, I am noticing that the AVCaptureDevice devices property is never updating. I have some minimal code to reproduce this. Run it with a USB camera plugged in, and as you unplug and replug the camera in the output never changes, and it says there are devices that don't actually exist. Note I am not running this in a UI application, so there is no standard loop. Its just meant to be running from a standard "c" main.#import <Foundation/Foundation.h> #import <AVFoundation/AVFoundation.h> int main(int argc, const char * argv[]) { @autoreleasepool { while (true) { NSArray* captureDevices = [AVCaptureDevice devicesWithMediaType:AVMediaTypeVideo]; for (id device in captureDevices) { NSString* uniqueIdentifier = [(AVCaptureDevice*)device uniqueID]; NSLog(uniqueIdentifier); } usleep(1000000); } } return 0; }Thanks for any help. I would be ok firing up a message loop in a background thread if I needed to, but I can't guarentee that the user application has a message loop (sorry if terminology is different, I come from windows).
Replies
4
Boosts
0
Views
1.1k
Activity
Dec ’21
aumf in logic pro x (10.4)
Hi, I've modified the AU Filter Demo to accept MIDI notes to control a parameter. I changed the type to aumf. It validates and loads into logic pro x 10.4 just fine.However, when I create a software instrument and go to insert my midi controlled effect in the input slot, it doesn't show up. I also notice that there aren't any other midi controlled effects in the plugin manager, so I don't have a reference case.Am I instantiating the audio unit correctly in logic pro (I couldn't find anything in the help docs for logic)? Is there something else I need to configure or set in the audio unit so that logic "sees" it correctly?Thanks!Chris
Replies
6
Boosts
0
Views
3.8k
Activity
Mar ’23
When playing video Content : KVO_IS_RETAINING_ALL_OBSERVERS_OF_THIS_OBJECTS_IF_IT_CRASHES_AN_OBSERVER_WAS_OVERRELEASED_OR_SMASHED
When i am playing video got this crash.Crashlytics log reportsCrashed: com.apple.main-thread0 libobjc.A.dylib 0x1845a67e8 object_isClass + 161 Foundation 0x185d103e8 KVO_IS_RETAINING_ALL_OBSERVERS_OF_THIS_OBJECT_IF_IT_CRASHES_AN_OBSERVER_WAS_OVERRELEASED_OR_SMASHED + 682 Foundation 0x185d0e8ec NSKeyValueWillChangeWithPerThreadPendingNotifications + 3003 AVFoundation 0x18abb60f8 -[AVPlayerItem willChangeValueForKey:] + 964 AVFoundation 0x18abc7ff0 -[AVPlayerItem _updatePropertyCacheAndTriggerKVOForPlayer:usingKeys:] + 2005 AVFoundation 0x18abb959c __60-[AVPlayerItem _informObserversAboutAvailabilityOfDuration:]_block_invoke_2 + 1846 libdispatch.dylib 0x184cdd088 _dispatch_call_block_and_release + 247 libdispatch.dylib 0x184cdd048 _dispatch_client_callout + 168 libdispatch.dylib 0x184d1ddfc _dispatch_main_queue_callback_4CF$VARIANT$armv81 + 9689 CoreFoundation 0x185301eb0 __CFRUNLOOP_IS_SERVICING_THE_MAIN_DISPATCH_QUEUE__ + 1210 CoreFoundation 0x1852ffa8c __CFRunLoopRun + 201211 CoreFoundation 0x18521ffb8 CFRunLoopRunSpecific + 43612 GraphicsServices 0x1870b7f84 GSEventRunModal + 10013 UIKit 0x18e7f42f4 UIApplicationMain + 20814 VZFiOSMobile 0x10475b340 main (main.m:17)15 libdyld.dylib 0x184d4256c start + 4Why does this happens and what I can do to avoid this?
Replies
9
Boosts
0
Views
8.8k
Activity
Nov ’21
(iOS) Issues with 'aumi' Audio Units and AVAudioEngine
Hi all,I just filled the bug report 39600781, regarding 'aumi' (kAudioUnitType_MIDIProcessor) audio units and AVAudioEngine. In short: in iOS, when attaching 'aumi' audio units to an AVAudioEngine, the AVAudioEngine will never call their renderBlock / internalRenderBlock, rendering them (pun 😝) useless.I write this post because I know that 'aumi' audio units in iOS are undocumented, thus some quirks are to be expected 🙂. But since some developers are already implementing them (I am beta testing their apps), I think it is important to raise awareness that 'aumi' audio units do not work with hosts that use AVAudioEngine. Note that hosts using AUGraph work, because they need to explicitly call the renderBlock method of their audio units. Yet since AUGraph is going to be deprecated... well, if "aumis" are to be officially introduced in the next iOS version, it is important to raise awareness about this particular issue. That's all. Thanks for reading 😉.
Replies
7
Boosts
0
Views
2.7k
Activity
Oct ’22
Switch camera while recording video?
Hi,I have an app that includes a very simple one-button interface for recording a video. Our customer wants to be able to switch between the front and rear cameras while the video is being recorded, and without any interruption in the video stream. I notice that even the iOS built-in camera app doesn't do this, but I've heard that some third-party apps do.Is this possible in a practical way? If so, how would I go about it?Thanks,Frank
Replies
3
Boosts
0
Views
15k
Activity
Nov ’21
Another photo orientation question
HiI'm using Swift 4 and the AvCam example code from Apple. In my own app I've an issue where the captured image orientation is wrong. My requirement is to save the photos to my applications folder as opposed to the Photos library. I've spent a day trying to correct my own code and got nowhere.As a test I modified the AvCam code to write the photo data to a static variable in a struct and then display that image on the camera view in AvCam by tapping a button as follows:// In the camera view controller: @IBOutlet weak var pv: UIImageView! // for previewing test @IBAction func btn(_ sender: UIButton) { // for setting preview test image let dataProvider = CGDataProvider(data: photoTest.photo! as CFData) let cgImageRef: CGImage! = CGImage(jpegDataProviderSource: dataProvider!, decode: nil, shouldInterpolate: true, intent: .defaultIntent) let photo = UIImage(cgImage: cgImageRef) pv.contentMode = .scaleAspectFill pv.image = photo }The orientaion issue is replicated in AvCam with the above code. In my own app the image is previewed before saving to the apps folder and the images orientation is always wrong in the preview as well as when reloading the image from disk. I.e. if the iphone is in portrait orientation, the preview image is rotated 90 degrees anti-clockwise, in landscape the same happens so the preview image appears to be the correct way up.I'm assuming I've completely missed the point somewhere along the line as it seems to me that regardless of device orientation, the top of a photo should always be the top. For example, in the Photos app photos are shown the correct way up regardless of orientation.Any help would be much appreciated, this is driving me nuts ;o)Cheers
Replies
5
Boosts
0
Views
4.5k
Activity
Feb ’22
Remote commands not delivered if AVPlayer is paused
I have a simple tvOS app that has a single audio track, a storyboard with single scene that contains a button for play/pause toggling.I am trying to add Siri Remote handling so that user can play/pause the track with remote's "toggle play/pause" button.The problem, though, is that when the application starts it doesn't respond to remote control events if the AVPlayer is paused. When user starts playback using the touch-button on Siri Remote I can pause the playback using "toggle play/pause" button. I CAN'T start playback again after the track has been paused.I tried two approaches:application.beginReceivingRemoteControlEvents()MPRemoteCommandCenter.shared().pauseCommand.addTarget(...)Using the remoteControlReceived(with event: UIEvent?) I receive the event only when the AVPlayer is playing and the event's subtype is pause command.Using the remote command center I registered for pauseCommand, playCommand and togglePlayPauseCommandcommands but only pauseCommand is being triggered when the playback is active.Am I missing something?
Replies
1
Boosts
0
Views
2.5k
Activity
Dec ’21
Implementing camera switch (front to back, vice versa)
I'm having trouble understand how to switch camera from front to back, back to front upon the touch of a button.My current code doesn't work.I get " Multiple audio/video AVCaptureInputs are not currently supported'Here is what I have so far:import UIKit import AVFoundation import Photos class CameraViewController :UIViewController { func applyRoundCorner(_ object: AnyObject){ object.layer.cornerRadius = (object.frame.size.width)/2 object.layer.masksToBounds = true } @IBOutlet weak var cameraButton: UIButton! @IBOutlet weak var imagePreview: UIImageView! @IBAction func Library(_ sender: Any) { } @IBAction func FlipCamera(_ sender: Any) { captureSession.stopRunning() cameraPreviewlayer?.removeFromSuperlayer() //cameraPreviewlayer = nil //self.captureSession = nil setupCaptureSession() if currentCamera! == backCamera{ //print(currentCamera) currentCamera = frontCamera} else { currentCamera = backCamera} setupCaptureSession() setupInputOutput() setupPreviewLayer() startRunningCaptureSession() } var captureSession = AVCaptureSession() var backCamera: AVCaptureDevice? var frontCamera: AVCaptureDevice? var currentCamera: AVCaptureDevice? var cameraPreviewlayer: AVCaptureVideoPreviewLayer? var photoOutput: AVCapturePhotoOutput? var image: UIImage? override func viewDidLoad() { super.viewDidLoad() self.applyRoundCorner(cameraButton) setupCaptureSession() setupInputOutput() setupPreviewLayer() startRunningCaptureSession() } func setupCaptureSession(){ captureSession.sessionPreset = AVCaptureSession.Preset.photo // Why exclamation point? let availableDevice = AVCaptureDevice.DiscoverySession(deviceTypes: [AVCaptureDevice.DeviceType.builtInWideAngleCamera], mediaType: .video, position: .unspecified) let devices = availableDevice.devices //back or front for device in devices { if device.position == AVCaptureDevice.Position.back{ backCamera = device }else if device.position == AVCaptureDevice.Position.front{ frontCamera = device } } currentCamera = backCamera } func setupInputOutput(){ do { let captureDeviceInput = try AVCaptureDeviceInput(device: currentCamera!) captureSession.addInput(captureDeviceInput) photoOutput = AVCapturePhotoOutput() photoOutput?.setPreparedPhotoSettingsArray([AVCapturePhotoSettings(format:[AVVideoCodecKey:AVVideoCodecType.jpeg])], completionHandler: nil) captureSession.addOutput(photoOutput!) } catch{ print(error) } } func setupPreviewLayer(){ cameraPreviewlayer = AVCaptureVideoPreviewLayer(session: captureSession) cameraPreviewlayer?.videoGravity = AVLayerVideoGravity.resizeAspectFill cameraPreviewlayer?.connection?.videoOrientation = AVCaptureVideoOrientation.portrait cameraPreviewlayer?.frame = self.imagePreview.frame self.view.layer.insertSublayer(cameraPreviewlayer!, at: 0) } func startRunningCaptureSession(){ captureSession.startRunning() } @IBAction func TakePhoto(_ sender: Any) { let settings = AVCapturePhotoSettings() photoOutput?.capturePhoto(with: settings, delegate: self) } override func prepare(for segue: UIStoryboardSegue, sender: Any?){ if segue.identifier == "showPhotoSegue"{ let previewVC = segue.destination as! PreviewViewController previewVC.image = self.image } } } extension CameraViewController: AVCapturePhotoCaptureDelegate{ func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto photo: AVCapturePhoto, error: Error?) { if let imageData = photo.fileDataRepresentation(){ image = UIImage(data: imageData) performSegue(withIdentifier:"showPhotoSegue", sender: nil) } } }
Replies
6
Boosts
0
Views
4k
Activity
Jan ’23
Audio renderer fails to render CMSampleBuffer
I am trying to render audio using AVSampleBufferAudioRenderer but there is no sound coming from my speakers and there is a repeated log message.[AQ] 405: SSP::Render: CopySlice returned 1I am creating a CMSampleBuffer from an AudioBufferList. This is the relevant code:var sampleBuffer: CMSampleBuffer! try runDarwin(CMSampleBufferCreate(allocator: kCFAllocatorDefault, dataBuffer: nil, dataReady: false, makeDataReadyCallback: nil, refcon: nil, formatDescription: formatDescription, sampleCount: sampleCount, sampleTimingEntryCount: 1, sampleTimingArray: &timingInfo, sampleSizeEntryCount: sampleSizeEntryCount, sampleSizeArray: sampleSizeArray, sampleBufferOut: &sampleBuffer)) try runDarwin(CMSampleBufferSetDataBufferFromAudioBufferList(sampleBuffer, blockBufferAllocator: kCFAllocatorDefault, blockBufferMemoryAllocator: kCFAllocatorDefault, flags: 0, bufferList: audioBufferList.unsafePointer)) try runDarwin(CMSampleBufferSetDataReady(sampleBuffer))I am pretty confident that my audio format description is correct because CMSampleBufferSetDataBufferFromAudioBufferList, which performs a laundry list of validations, returns no error.I tried to reverse-engineer the CopySlice function, but I’m lost without the parameter names.int ScheduledSlicePlayer::CopySlice( long long, ScheduledSlicePlayer::XScheduledAudioSlice*, int, AudioBufferList&, int, int, bool )Does anyone have any ideas on what’s wrong? For the Apple engineers reading this, can you tell me the parameter names of the CopySlice function so that I can more easily reverse-engineer the function to see what the problem is?
Replies
4
Boosts
1
Views
3.3k
Activity
Oct ’22
AudioComponentInstanceNew takes several seconds to complete
I'm using a VoiceProcessingIO audio unit in my VoIP application on Mac. The problem is, at least since Mojave, AudioComponentInstanceNew blocks for at least 2 seconds. Profiling shows that internally it's waiting on some mutex and then on some message queue. My code to initialize the audio unit is as follows: OSStatus status; AudioComponentDescription desc; AudioComponent inputComponent; desc.componentType = kAudioUnitType_Output; desc.componentSubType = kAudioUnitSubType_VoiceProcessingIO; desc.componentFlags = 0; desc.componentFlagsMask = 0; desc.componentManufacturer = kAudioUnitManufacturer_Apple; inputComponent = AudioComponentFindNext(NULL, &desc); status = AudioComponentInstanceNew(inputComponent, &unit);Here's a profiler screenshot showing the two system calls in question.So, is this a bug or an intended behavior?
Replies
4
Boosts
0
Views
2.6k
Activity
Jul ’23
double output of a camera video stream
Hello, everybody,I would like to display the camera image of the back camera twice on the screen (as with the cardboard). I created two views and they each take 50% of the screen, I can also output the live image of the camera on one view.// views left and right var viewRect:CGRect! var leftView:UIView! var rightView:UIView! //... //left viewRect = CGRect(x: 0, y: 0, width: 0.5 * self.view.frame.width, height: self.view.frame.height) leftView = UIView(frame: viewRect) leftView.backgroundColor = .green view.addSubview(leftView) //rgiht viewRect = CGRect(x: 0.5 * self.view.frame.width, y: 0, width: 0.5 * self.view.frame.width, height: self.view.frame.height) rightView = UIView(frame: viewRect) rightView.backgroundColor = .red view.addSubview(rightView) //... func beginSession(){ captureSession.sessionPreset = AVCaptureSession.Preset.photo let devices = AVCaptureDevice.devices() // Loop through all the capture devices on this phone for device in devices { // Make sure this particular device supports video if (device.hasMediaType(AVMediaType.video)) { // Finally check the position and confirm we've got the back camera if(device.position == AVCaptureDevice.Position.back) { captureDevice = device as? AVCaptureDevice } } } do { try captureSession.addInput(AVCaptureDeviceInput(device: captureDevice!)) captureSession.sessionPreset = AVCaptureSession.Preset.photo } catch _ { } let previewLayer = AVCaptureVideoPreviewLayer(session: captureSession) previewLayer.connection?.videoOrientation = AVCaptureVideoOrientation.landscapeLeft previewLayer.videoGravity = AVLayerVideoGravity.resizeAspectFill self.leftPreviewLayer = previewLayer self.leftPreviewLayer.frame = self.view.layer.frame self.leftPreviewLayer.frame = self.leftView.bounds self.leftView.layer.addSublayer(previewLayer) self.leftView.layer.addSublayer(previewLayer) captureSession.startRunning() }I already found something like "let replicatorLayer = CAReplicatorLayer()" online, but can't get it to work 😟it would be great if someone could give me a hint 😉Thx and cya Tom
Replies
1
Boosts
0
Views
1.2k
Activity
Apr ’22
When will AVReaderWriter example be updated?
The AVReaderWriter was last updated in 2016 for Swift 2.3. I am a newby, and have been unable to upgrade or convert it using the Xcode tools. I tried going from Swift 2 to 3, in preparation of going from 3 to 4, but got a ton of errors. I downloaded every version of Xcode since 8.0 to try and incrementally upgrade it, and it quickly fell apart. Is there an update schedule for their documentation? I hear Swift 5 is coming out, and I am sure 6 and 7 are not far behind. I'm not sure how people are supposed to figure this stuff out if the documentation is several versions out-of-date, I guess it is easier if you have a lot of experience.
Replies
11
Boosts
0
Views
3.1k
Activity
Dec ’21
AVAudioEngine returning silence at the beginning of manual rendering
I am using AVAudioEngine to analyse some audio tracks and quite often the `while engine.manualRenderingSampleTime < sourceFile.length {` loop takes a moment to start receiving audio data. Looking at the soundwave looks like the input is simply delayed. This wouldn't be a problem if the length of the buffer would vary depending on this latency but the length unfortunately stays always the same, loosing the final part of the track.I took the code from the official tutorial and this seems to happen regardless if I add an EQ effect or not. Actually looks like the two analysis (with or without EQ) done one after the other return the same anomaly. let format: AVAudioFormat = sourceFile.processingFormat let engine = AVAudioEngine() let player = AVAudioPlayerNode() engine.attach(player) if compress { let eq = AVAudioUnitEQ(numberOfBands: 2) engine.attach(eq) let lowPass = eq.bands[0] lowPass.filterType = .lowPass lowPass.frequency = 150.0 lowPass.bypass = false let highPass = eq.bands[1] highPass.filterType = .highPass highPass.frequency = 100.0 highPass.bypass = false engine.connect(player, to: eq, format: format) engine.connect(eq, to: engine.mainMixerNode, format: format) }else{ engine.connect(player, to: engine.mainMixerNode, format: format) } do { let maxNumberOfFrames: AVAudioFrameCount = 4096 try engine.enableManualRenderingMode(.offline, format: format, maximumFrameCount: maxNumberOfFrames) } catch { fatalError("Could not enable manual rendering mode, \(error)") } player.scheduleFile(sourceFile, at: nil) do { try engine.start() player.play() }catch{ fatalError("Could not start engine, \(error)") } // buffer to which the engine will render the processed data let buffer: AVAudioPCMBuffer = AVAudioPCMBuffer(pcmFormat: engine.manualRenderingFormat, frameCapacity: engine.manualRenderingMaximumFrameCount)! // var pi = 0 // while engine.manualRenderingSampleTime < sourceFile.length { do { let framesToRender = min(buffer.frameCapacity, AVAudioFrameCount(sourceFile.length - engine.manualRenderingSampleTime)) let status = try engine.renderOffline(framesToRender, to: buffer) switch status { case .success: // data rendered successfully let flength = Int(buffer.frameLength) points.reserveCapacity(pi + flength) if let chans = buffer.floatChannelData?.pointee { let left = chans.advanced(by: 0) let right = chans.advanced(by: 1) for b in 0..<flength { let v:Float = max(abs(left[b]), abs(right[b])) points.append(v) } } pi += flength case .insufficientDataFromInputNode: // applicable only if using the input node as one of the sources break case .cannotDoInCurrentContext: // engine could not render in the current render call, retry in next iteration break case .error: // error occurred while rendering fatalError("render failed") } } catch { fatalError("render failed, \(error)") } } player.stop() engine.stop()I thought it was perhaps a simulator issue, but also on the device is happening. Am I doing anything wrong? Thanks!
Replies
2
Boosts
0
Views
2.9k
Activity
Oct ’21
When Launching Camera App Crashes
I am trying to access the camera to take pictures in my app. Here is my code:@IBAction func uploadPicture(_ sender: Any) { imagePickerController = UIImagePickerController() imagePickerController.delegate = self imagePickerController.sourceType = .camera present(imagePickerController, animated: true, completion: nil) } func saveImage(imageName: String){ //create an instance of the FileManager let fileManager = FileManager.default //get the image path let imagePath = (NSSearchPathForDirectoriesInDomains(.documentDirectory, .userDomainMask, true)[0] as NSString).appendingPathComponent(imageName) //get the image we took with camera if pic4 != nil { let image = pic4 //get the PNG data for this image let data = image.jpegData(compressionQuality: 1) //store it in the document directory fileManager.createFile(atPath: imagePath as String, contents: data, attributes: nil) // print("Big Bobs\(image.imageOrientation.rawValue)") if image.size.height > image.size.width { newWine.isPortrait = true } print("Bigmacs \(image.imageOrientation.rawValue)") } } func imagePickerController(_ picker: UIImagePickerController, didFinishPickingMediaWithInfo info: [UIImagePickerController.InfoKey : Any]) { //// Local variable inserted by Swift 4.2 migrator. //let info = convertFromUIImagePickerControllerInfoKeyDictionary(info) // // // if let image2 = info[convertFromUIImagePickerControllerInfoKey(UIImagePickerController.InfoKey.originalImage)] as? UIImage { // // newWine.picture = image2 // // // } // // self.dismiss(animated: true, completion: nil) imagePickerController.dismiss(animated: true, completion: nil) // makes image show in image view if I want pic4 = (info[UIImagePickerController.InfoKey.originalImage] as? UIImage)! // imageView.image = info[UIImagePickerController.InfoKey.originalImage] as? UIImage SVProgressHUD.showSuccess(withStatus: "Picture Added Successfully") newWine.pictureIncluded = true }However when I press the upLoadPicture button the app crashes. Everything works fine when using the .photoLibrary in line 5.My info.plist includes the following:Privacy - Photo Library Additions Usage DescriptionPrivacy - Photo Library Usage DescriptionPrivacy  -  Camera Usage DescriptionThanks to all help!
Replies
5
Boosts
0
Views
4.1k
Activity
Aug ’22
How to specify bit rate when writing with AVAudioFile
I can currently write, using AVAudioFile, to any of the file formats specified by Core Audio.It can create files in all formats (except one, see below ) that can be read into iTunes, Quicktime and other apps and played back.However some formats appear to be ignoring values in the AVAudioFile settings dictionary.e.g: • An MP4 or AAC will save and write successfully at any sample rate but any bit rates I attempt to specify are ignored. • Wave files saved with floating point data are always converted to Int32 even though I specify float. Even though the PCM buffers I’m using as input and output for sample rate conversion are float on input and output. So the AVAudioFile is taking Float input but converting it to Int for some reason I can’t fathom. • The only crash/exception/failure I see is if I attempt to create an AVAudioFile as WAV/64 bit float. … bang, AVAudioFile isn’t having that one!The technique I’m using is: • Create AVAudioFile for writing with a settings dictionary. • Get processing and file format from AVAudioFile • Client format is always 32 bit Float, AVAudioFile generally reports its processing format as some other word sized Float format at the sample rate and size I’ve specified in the fileFormat. • Create a converter to convert from client format to processing format. • Process input data through the converter to the file using converter.convert(to: , error:&error, withInputFrom )So this works … sort ofThe files ( be they wav, aiff, flac, pp3, aac, mp4 etc ) are written out and will play back just fine.… but …If the processing word format is Float, in a PCM file like WAV, the AVAudioFile will always report its fileFormat as Int32.And if the file is a compressed format such as mp4/aac, any bit rates I attempt to specify are just ignored but the sample rate appears to be respected as if the converters/encoders just choose a bit rates based on sample rate.So after all that waffle, I've missed something that's probably meant to be obvious, so my questions are … • For lpcm float formats why is Int32 data written even though the AVAudioFile settings dictionary has AVLinearPCMIsFloatKey to true ? • How do arrange the setup so that I can specify the bit rate for compressed audio?The only buffers I actually create are both PCM, the client output buffer, and the AVAudioConverter/AVAudioFile processing buffer.I’ve attempted using AVAudioCompressedBuffer but haven’t had any luck.I hope someone has some clues because I’ve spent more hours on this than anyone should ever need to!For my Christmas present I’d like Core Audio to be fully and comprehensively documented please!
Replies
4
Boosts
0
Views
4.9k
Activity
Aug ’21
UVC Camera ,AVFoundation can not start video stream
I develop a application with an uvc camera, this camera is a webcam, I use the AVFoundation library ,but when I run the code "[self.mCaptureSession startRunning]" ,I can not get the buffer, I already set the delegate, any answer will help.
Replies
1
Boosts
0
Views
1.3k
Activity
Dec ’25
Why does Apple use simplified 1.961 gamma instead of precise ITU-R 709 transfer function?
(For Apple folks: rdar://47577096.)# BackgroundThe Core Video function `CVImageBufferCreateColorSpaceFromAttachments` creates custom color profiles with simplified transfer functions instead of using the standard system color profiles. Let’s take ITU-R 709 as an example.The macOS `Rec. ITU-R BT.709-5` system color profile specifies the transfer function asf(x) = { (0.91x + 0.09)^2.222 where x >= 0.081 { 0.222x where x < 0.081The Apple-custom `HDTV` color profile created by the above Core Video function specifies the transfer function asf(x) = x^1.961My understanding is that `x^1.961` is the closest approximation of the more complex ITU-R 709 transfer function.# Questions1. Why use a custom color profile with a simplified transfer function rather than the official specification? - Was it done for performance? - Was it done for compatibility with non-QuickTime-based applications? - etc.2. Speaking of compatibility, there is a problem when an encoding application uses the official transfer function and the decoding application uses the approximated transfer function. I tested this using two images. One image uses the `Rec. ITU-R BT.709-5` color profile. The other image is derived from the former by assigning the Apple-custom `HDTV` color profile. The latter image loses the details in the darker areas of the image. Why go to the trouble of approximating the transfer function when the approximation isn’t that great?3. Are the Apple-custom color profiles also used for encoding? Or are they only for decoding?4. Another thing that concerns me is that the Apple-custom `HDR (PQ)` and `HDR (HLG)` color profiles use the same simplified transfer function of `f(x) = x^1.801`. Isn’t the whole point of the PQ and HLG standards to define more sophisticated transfer functions? Doesn’t simplifying those two transfer functions defeat their purpose?
Replies
1
Boosts
1
Views
3.8k
Activity
Apr ’22
Trouble getting good bluetooth audio behavior
We are trying to get good bluetooth behavior for our app but there are a whole slew of problems.There are 2 issues that I don't understand how to resolve,1. I don't receive notifications from AVAudioSessionRouteChangeNotification if user starts app without Bluetooth headphones connected.2. If I stream with Apple Music, after adding the code to enable Bluetooth, all audio from my app is lost unless I put app in background then bring it to the foreground at which the audio is now forced into the earpiece speaker phoneWithout making any changes to the code to try and incorporate bluetooth capabilities our app currently forces audio through the speakers.To enable bluetooth we've added the following code just before the app finishes initializing which works great if the user already has their bluetooth headphones connected. The audio will successfully be outputed into the headphones. If they disconnect their headphones the audio will be outputted to the device speakers.[[AVAudioSession sharedInstance] setCategory:AVAudioSessionCategoryPlayAndRecord withOptions:AVAudioSessionCategoryOptionDefaultToSpeaker | AVAudioSessionCategoryOptionAllowBluetoothA2DP | AVAudioSessionCategoryOptionAllowAirPlay error:nil]; [[AVAudioSession sharedInstance] setActive:YES error:nil];However if the user is streaming music through our app then behavior becomes very poor. We have 2 ways of streaming music, the first is using an AVPlayer to stream music from a URL, the other way is using MPMusicPlayerController to stream music from Apple Music.After adding the code to enable bluetooth wireless headphones, if the user starts streaming music with Apple Music, all audio such as sound effects from our app get turned off and the only thing the user can hear is the music being streamed. If they stop streaming the music the audio never comes back. If they then turn off the bluetooth wireless headphones and put the app into the background followed by bringing it back into the foreground, the sound effects come back but now they are outputted through the earphone speaker.That is the worst of all the behavior we see.When streaming with the AVPlayer we don't see any issues with the app sound effects being lost but, if the user disconnects their bluetooth wireless headphones while streaming I cannot get the app to detect when they become available again if the user turns them on. I have the following code to listen when the audio routes change and that works great when disconnecting the bluetooth wireless headphones, but when reconnecting them I can't get a notification unless after I've turned my bluetooth headphones on that I put the app into the background and bring it back into the foreground.NSNotificationCenter* notificationCenter = [NSNotificationCenter defaultCenter]; [notificationCenter addObserver:media_sharedInstance selector:@selector(audioHardwareRouteChanged:) name:AVAudioSessionRouteChangeNotification object:nil];- (void) audioHardwareRouteChanged:(NSNotification *)notification{ NSInteger routeChangeReason = [notification.userInfo[AVAudioSessionRouteChangeReasonKey] integerValue]; NSLog(@"route change"); NSLog(@"%ld",(long)routeChangeReason); NSLog(@"output volume %f",[[AVAudioSession sharedInstance] outputVolume]); if(routeChangeReason == AVAudioSessionRouteChangeReasonOldDeviceUnavailable) { //[media_sharedInstance RemoveAudioChange]; NSLog(@"old device unavailable"); [media_sharedInstance ConfigureSpeakers]; //[media_sharedInstance SubscribeToAudioChange]; } if(routeChangeReason == kAudioSessionRouteChangeReason_NewDeviceAvailable) { NSLog(@"new device available"); //[media_sharedInstance RemoveAudioChange]; [media_sharedInstance ConfigureSpeakers]; //[media_sharedInstance SubscribeToAudioChange]; } } - (void) ConfigureSpeakers { [[AVAudioSession sharedInstance] setCategory:AVAudioSessionCategoryPlayAndRecord withOptions:AVAudioSessionCategoryOptionDefaultToSpeaker | AVAudioSessionCategoryOptionAllowBluetooth | AVAudioSessionCategoryOptionAllowAirPlay error:nil]; [[AVAudioSession sharedInstance] setActive:YES error:nil]; }Edit* I figured out the issue with AVAudioSessionRouteChangeNotification not sending a notification. The solution was to use AVAudioSessionCategoryOptionAllowBluetoothA2DP and AVAudioSessionCategoryOptionAllowBluetooth when I received the notification that the bluetooth was disconnected, and when I get the notification that bluetooth becomes available I switch the options to just AVAudioSessionCategoryOptionAllowBluetoothA2DPStill trying to figure out why all of the audio from the app gets shut down when playing Apple Music (MPMusicPlayerController)
Replies
1
Boosts
0
Views
2.3k
Activity
Sep ’22