How to pass AR Session's frame to Vision? In SwiftUI

Question

fjybiocs OP

Created Sep ’22

Replies 1

Boosts 0

Views 1.6k

Participants 2

I'm making an app:

It displays the image of the rear camera in real time. (in AR View)（✅）
When it detects a QR code, it will read the text content in the QR code.（I don't know how to do it in real time, get data from AR session）

func getQRCodeContent(_ pixel: CVPixelBuffer) -> String {
    let requestHandler = VNImageRequestHandler(cvPixelBuffer: pixel, options: [:])
    let request = VNDetectBarcodesRequest()
    request.symbologies = [.qr]

    try! requestHandler.perform([request])
    let result = request.results?.first?.payloadStringValue
    
    if let result = result {
        return result
    } else {
        return "non"
    }
}

And then do some logic with the content, and display the corresponding AR model in the AR View.

I know I have to feed images into the Vision Framework, I started with AVFoundation, but I found that when AR View is loaded, the AVCaptureSession is paused.

And I want to feed AR Session's frame into Vision Framework. However, all the tutorials I can find are based on story board and UI kit to complete this function. I don't know how to complete this function in Swift UI at all.

I tried to extent ARView:

extension ARView: ARSessionDelegate {
    func renderer(_ renderer: SKRenderer, willRenderScene scene: SCNScene, atTime time: TimeInterval) {
        let capturedImage = session.currentFrame?.capturedImage
        print(capturedImage)
    }
}

struct ARViewCustom: UIViewRepresentable {
    func makeUIView(context: Context) -> ARView {
        let arView = ARView(frame: .zero)
        arView.session.delegate = arView
        return arView
    }

    func updateUIView(_ uiView: ARView, context: Context) {}
}

No error, but it doesn't work.

Boost

Answer 1

DTS Engineer OP

Apple

Sep ’22

Hello,

Typically when you need a delegate object in a SwiftUI representable, you use a Coordinator, for example:

struct ARViewContainer: UIViewRepresentable {
    
    func makeUIView(context: Context) -> ARView {
        
        let arView = ARView(frame: .zero)
        
        arView.session.delegate = context.coordinator
        
        return arView
        
    }
    
    func updateUIView(_ uiView: ARView, context: Context) {}
    
    func makeCoordinator() -> Coordinator {
        Coordinator()
    }
    
    final class Coordinator: NSObject, ARSessionDelegate {
        
        let request = VNDetectBarcodesRequest()
        
        let requestQueue = DispatchQueue(label: "Request Queue")
        
        var frameCount = 0
        
        var shouldRunBarcodeDetection: Bool {
            // True every ten frames.
            frameCount % 10 == 0
        }
        
        func session(_ session: ARSession, didUpdate frame: ARFrame) {
            
            frameCount &+= 1
            
            // You probably don't need to run barcode detection *every* frame, that could get pretty expensive.
            if shouldRunBarcodeDetection {
                detectBarcodes(in: frame)
            }
        }
        
        private func detectBarcodes(in frame: ARFrame) {
            // Perform the request off the main queue to avoid blocking the UI (and the ARSessionDelegate, who's methods are by default called on the main queue).
            requestQueue.async { [unowned self] in
                let requestHandler = VNImageRequestHandler(cvPixelBuffer: frame.capturedImage)
                
                do {
                    try requestHandler.perform([request])
                    
                    if let results = request.results {
                        for result in results {
                            print(result)
                        }
                    }
                } catch {
                    fatalError(error.localizedDescription)
                }
            }
        }
    }
}

0