Unexpected Behavior in Entity Movement System When Using AVAudioPlayer in visionOS Development

I am currently developing an app for visionOS and have encountered an issue involving a component and system that moves an entity up and down within a specific Y-axis range. The system works as expected until I introduce sound playback using AVAudioPlayer.

Whenever I use AVAudioPlayer to play sound, the entity exhibits unexpected behaviors, such as freezing or becoming unresponsive. The freezing of the entity's movement is particularly noticeable when playing the audio for the first time. After that, it becomes less noticeable, but you can still feel it, especially when the audio is played in quick succession.

Also, the issue is more noticable on real device than the simulator

//
//  IssueApp.swift
//  Issue
//
//  Created by Zhendong Chen on 2/1/25.
//

import SwiftUI

@main
struct IssueApp: App {
    var body: some Scene {
        WindowGroup {
            ContentView()
        }
        .windowStyle(.volumetric)
    }
}
//
//  ContentView.swift
//  Issue
//
//  Created by Zhendong Chen on 2/1/25.
//

import SwiftUI
import RealityKit
import RealityKitContent

struct ContentView: View {

    @State var enlarge = false

    var body: some View {
        RealityView { content, attachments in
            // Add the initial RealityKit content
            if let scene = try? await Entity(named: "Scene", in: realityKitContentBundle) {
                if let sphere = scene.findEntity(named: "Sphere") {
                    sphere.components.set(UpAndDownComponent(speed: 0.03, minY: -0.05, maxY: 0.05))
                }
                if let button = attachments.entity(for: "Button") {
                    button.position.y -= 0.3
                    scene.addChild(button)
                    
                }
                content.add(scene)
            }
        } attachments: {
            Attachment(id: "Button") {
                VStack {
                    Button {
                        SoundManager.instance.playSound(filePath: "apple_en")
                    } label: {
                        Text("Play audio")
                    }
                    .animation(.none, value: 0)
                    .fontWeight(.semibold)
                }
                .padding()
                .glassBackgroundEffect()
            }
        }
        .onAppear {
            UpAndDownSystem.registerSystem()
        }
    }
}
//
//  SoundManager.swift
//  LinguaBubble
//
//  Created by Zhendong Chen on 1/14/25.
//

import Foundation
import AVFoundation


class SoundManager {
    static let instance = SoundManager()
    
    private var audioPlayer: AVAudioPlayer?
    
    func playSound(filePath: String) {
        guard let url = Bundle.main.url(forResource: filePath, withExtension: ".mp3") else { return }
        
        do {
            audioPlayer = try AVAudioPlayer(contentsOf: url)
            audioPlayer?.play()
        } catch let error {
            print("Error playing sound. \(error.localizedDescription)")
        }
        
    }


}
//
//  UpAndDownComponent+System.swift
//  Issue
//
//  Created by Zhendong Chen on 2/1/25.
//

import RealityKit

struct UpAndDownComponent: Component {
    var speed: Float
    var axis: SIMD3<Float>
    var minY: Float
    var maxY: Float
    var direction: Float = 1.0 // 1 for up, -1 for down
    var initialY: Float?
    
    init(speed: Float = 1.0, axis: SIMD3<Float> = [0, 1, 0], minY: Float = 0.0, maxY: Float = 1.0) {
        self.speed = speed
        self.axis = axis
        self.minY = minY
        self.maxY = maxY
    }
}



struct UpAndDownSystem: System {
    static let query = EntityQuery(where: .has(UpAndDownComponent.self))
    
    init(scene: RealityKit.Scene) {}

    func update(context: SceneUpdateContext) {
        let deltaTime = Float(context.deltaTime) // Time between frames
        
        for entity in context.entities(matching: Self.query, updatingSystemWhen: .rendering) {
            guard var component: UpAndDownComponent = entity.components[UpAndDownComponent.self] else { continue }

            // Ensure we have the initial Y value set
            if component.initialY == nil {
                component.initialY = entity.transform.translation.y
            }
            
            // Calculate the current position
            let currentY = entity.transform.translation.y
            
            // Move the entity up or down
            let newY = currentY + (component.speed * component.direction * deltaTime)
            
            // If the entity moves out of the allowed range, reverse the direction
            if newY >= component.initialY! + component.maxY {
                component.direction = -1.0 // Move down
            } else if newY <= component.initialY! + component.minY {
                component.direction = 1.0 // Move up
            }

            // Apply the new position
            entity.transform.translation = SIMD3<Float>(entity.transform.translation.x, newY, entity.transform.translation.z)
            
            // Update the component with the new direction
            entity.components[UpAndDownComponent.self] = component
        }
    }
}

Could someone help me with this?

Answered by Vision Pro Engineer in 823611022

Hello @WeAreAllSatoshi

Thank you for your detailed write up! It was very helpful and I was able to reproduce the issue you’re seeing.



My first suggestion is to cache your instances of AVAudioPlayer. In the code you provided, you create a new instance of AVAudioPlayer for the same audio file each time the button is pressed. This is not ideal. Here’s an edited version of your SoundManager that uses a dictionary to store references to existing instances of AVAudioPlayer, so that it only ever gets created once:




class SoundManager {
    static let instance = SoundManager()
    
    private var audioPlayer: AVAudioPlayer?
    private var audioPlayerCache: [String:AVAudioPlayer] = .init()
    
    func playSound(filePath: String) {
        guard let url = Bundle.main.url(forResource: filePath, withExtension: ".mp3") else { return }
        do {
            var audioPlayer: AVAudioPlayer
            if let existingPlayer = audioPlayerCache[filePath] {
                audioPlayer = existingPlayer
            } else {
                audioPlayer = try AVAudioPlayer(contentsOf: url)
                audioPlayerCache[filePath] = audioPlayer
            }
            audioPlayer.play()
        } catch let error {
            print("Error playing sound. \(error.localizedDescription)")
        }
    }
}


Alternatively, because you are using a RealityView and ECS, you may want to consider the .playAudio(_ resource: AudioResource) API to play audio from an Entity. Note that by default this will play spatial audio with reverb added, but this can also be configured depending on the type of component used to play back audio. Here’s an example function you could add to SoundManager: 



func playSoundOnEntity(entity: Entity, filePath: String) async {
    guard let url = Bundle.main.url(forResource: filePath, withExtension: ".mp3") else { return }
    do {
        let audioResource: AudioFileResource = try await .init(contentsOf: url)
        await entity.playAudio(audioResource)
    } catch {
        print("Error playing sound. \(error.localizedDescription)")
    }
}

Both of the solutions I suggested will solve the performance issues you are seeing in the current implementation. Let me know if that helped, and please reach out with any more questions you have!

Accepted Answer

Hello @WeAreAllSatoshi

Thank you for your detailed write up! It was very helpful and I was able to reproduce the issue you’re seeing.



My first suggestion is to cache your instances of AVAudioPlayer. In the code you provided, you create a new instance of AVAudioPlayer for the same audio file each time the button is pressed. This is not ideal. Here’s an edited version of your SoundManager that uses a dictionary to store references to existing instances of AVAudioPlayer, so that it only ever gets created once:




class SoundManager {
    static let instance = SoundManager()
    
    private var audioPlayer: AVAudioPlayer?
    private var audioPlayerCache: [String:AVAudioPlayer] = .init()
    
    func playSound(filePath: String) {
        guard let url = Bundle.main.url(forResource: filePath, withExtension: ".mp3") else { return }
        do {
            var audioPlayer: AVAudioPlayer
            if let existingPlayer = audioPlayerCache[filePath] {
                audioPlayer = existingPlayer
            } else {
                audioPlayer = try AVAudioPlayer(contentsOf: url)
                audioPlayerCache[filePath] = audioPlayer
            }
            audioPlayer.play()
        } catch let error {
            print("Error playing sound. \(error.localizedDescription)")
        }
    }
}


Alternatively, because you are using a RealityView and ECS, you may want to consider the .playAudio(_ resource: AudioResource) API to play audio from an Entity. Note that by default this will play spatial audio with reverb added, but this can also be configured depending on the type of component used to play back audio. Here’s an example function you could add to SoundManager: 



func playSoundOnEntity(entity: Entity, filePath: String) async {
    guard let url = Bundle.main.url(forResource: filePath, withExtension: ".mp3") else { return }
    do {
        let audioResource: AudioFileResource = try await .init(contentsOf: url)
        await entity.playAudio(audioResource)
    } catch {
        print("Error playing sound. \(error.localizedDescription)")
    }
}

Both of the solutions I suggested will solve the performance issues you are seeing in the current implementation. Let me know if that helped, and please reach out with any more questions you have!

Thank you so much for the solutions. The second worked the best for me.

Unexpected Behavior in Entity Movement System When Using AVAudioPlayer in visionOS Development
 
 
Q