HEVC Video with Alpha in SwiftUI

Question

After stumbling upon this video from WWDC '19, I have been trying to get transparent HEVC (H.265) videos to play in SwiftUI. Unfortunately I have been unable to get it working. Is this truly supported in iOS and tvOS 13+?

Here is my code:


import AVKit

import SceneKit

import SpriteKit



struct VideoView: View {

    @State var videoPlayer = AVPlayer(url:  Bundle.main.url(forResource: "fl", withExtension: "mov")!)

    @State var mainScene = SKScene(size: CGSize(width: 500, height: 500))

    

    var body: some View {

        HStack {

            Spacer()

            

            VStack {

                

                Spacer()

                SpriteView(scene: mainScene)

                

                Spacer()

                

            }.onAppear {

                guard let scene = SKScene(fileNamed: "backgroundScene") else {

                    print ("Could not create a background scene")

                    return

                }

                

                scene.scaleMode = .aspectFill

                scene.backgroundColor = .blue

                scene.view?.allowsTransparency = true

                guard let alphaMovieURL = Bundle.main.url(forResource: "output", withExtension: "mov") else {

                    print("Failed to overlay alpha movie on the background")

                    return

                }

                

                

                videoPlayer = AVPlayer(url: alphaMovieURL)

                

                let video = SKVideoNode(avPlayer: videoPlayer)

                video.size = CGSize(width: 500, height: 500)

                scene.addChild(video)

                videoPlayer.play()

                mainScene = scene

            }

            Spacer()

        }.background(Color.red)

    }

}

The file I have been using can be found here:

https://firebasestorage.googleapis.com/v0/b/leaderboard-d5992.appspot.com/o/fl.mov?alt=media&token=c939cdfc-5047-4c86-a38d-89d5a0f3c459

It very well could be that the encoding is wrong, although I have checked it many times and it definitely contains an alpha layer.

Any direction or pointers would be greatly appreciated!

166

Posted by

nparcher24

Reply

Add a Comment

Answer 1

Also, for clarification, this works but contains a black background. The desired output would have a transparent background :)

Posted by

nparcher24

Add a Comment

Answer 2

This appears to be working for me on macOS and iOS at least (didn't check tvOS, but suspect it works).

I slightly modified your code:

struct VideoView: View {

    @State var mainScene = SKScene()

    var body: some View {

        HStack {
            Spacer()
            VStack {
                Spacer()
                SpriteView(scene: mainScene)
                Spacer()
            }.onAppear {
                let scene = SKScene(size: .init(width: 500, height: 500))
                scene.scaleMode = .aspectFit
                scene.backgroundColor = .blue
                scene.view?.allowsTransparency = true
                
                let player = AVPlayer(url:  Bundle.main.url(forResource: "output", withExtension: "mov")!)


                let video = SKVideoNode(avPlayer: player)
                video.size = CGSize(width: 500, height: 500)
                video.anchorPoint = .init(x: 0, y: 0)
                scene.addChild(video)
                player.play()
                mainScene = scene
            }
            Spacer()
        }.background(Color.red)
    }
}

I end up seeing the video play, and the background color is whatever the scene's background color is (in this case, blue), which is the expected behavior.

Posted by

gchiste

@gchiste, can you confirm how you got this working? I have tried this code in many different sample projects but cannot get it working. Are you using an Intel Mac or ARM Mac? What version of Xcode / simulator software? I'm ultimately trying to get this working on tvOS so I'm currently limited to the simulator.

—
nparcher24
It's possible that this is an issue only in the Simulator, try seeing if it works on an actual device. I tested on an M1 Mac mini (macOS Ventura beta 2) and an iPhone 13 Pro (iOS 16 beta 2).

—
gchiste

Add a Comment

Answer 3

Thanks for the quick reply. It appears there are other issues at play here in my system.

The above code runs on the tvOS simulator but has a black background still. iOS and macOS both spit strange errors (of which I'm researching at the moment):

'-[MTLDebugRenderCommandEncoder validateCommonDrawErrors:]:5252: failed assertion `Draw Errors Validation

Fragment Function(FastSingle_FragFunc): missing sampler binding at index 0 for u_texture_sampler[0].'

For simplicity sake I made a GitHub repo here:

https://github.com/nparcher24/test-transparency.git

Does anyone else see something missing here? Is it my machine having this issue?

Posted by

nparcher24

Add a Comment

HEVC Video with Alpha in SwiftUI

Replies