What's new in Apple Immersive Video
Explore the latest updates in Apple Immersive Video tools, resources, and related technologies.
Apple Immersive Live formats
Live sports and events demand more than coverage. They demand presence. Livestreaming Apple Immersive Video delivers exactly that, placing every contested play, every roar of the crowd, and every moment of drama at lifelike acuity and scale inside Apple Vision Pro. Live Media Formats and tools bring native Apple Immersive Video workflows directly into a professional broadcast infrastructure that's built on SMPTE ST 2110, transporting ProRes video, Spatial Audio, and calibrated lens metadata together before being encoded to MV-HEVC. Explore the complete pipeline for live immersive production.
Immersive Media Support framework
The Immersive Media Support (IMS) framework gives you everything you need to read, write, and author rich metadata for Apple Immersive Video. Already available on visionOS and macOS, IMS now extends to iPadOS and iOS, bringing processing, control, and playback of immersive content to iPhone and iPad with the same cross-platform API.
Take control of camera parameters in live and complex production scenarios, even when they weren't part of the original venue description. New camera override capabilities let you define a camera inline, extending or replacing static metadata as your production evolves. Reference an existing camera or introduce a new one, and specify rotation, the calibration model, and an optional mask derived from AIME data. Overrides are honored at render time for the frame, giving you frame-accurate control over how immersive content is captured and presented.
With visionOS 27 handle the presentation of media shot from cameras mounted in any orientation with new Y-axis flipping, in addition to X-axis flipping. The flip is applied with full stereo awareness during playback, ensuring both eyes are correctly handled as a stereo pair, so footage from inverted camera rigs integrates seamlessly into your immersive productions.
Static foveation for Apple Immersive Video distribution
Apple Immersive Video (AIV) is captured at 8K per eye at 90 frames per second, generating over 10 billion pixels per second. Streaming that volume of data over even the fastest available networks isn't practical, but scaling the image down to 4K can sacrifice the pixel density that gives immersive content its sense of presence. Instead, creators can now apply a static foveation distribution function that dedicates most of the smaller frame to the pixels of primary importance, typically the central area of the fisheye lens. The result is a streamable 4K frame size that preserves most of the perceptual resolution at the original 90 fps, and it's one of the reasons Apple Immersive Video achieves such high perceptual acuity at easily streamable file sizes. A new sample project demonstrates how this technique can be used with IMS to deliver high-acuity immersive video in a streamable frame size.
Unique-sized portal views
Watching Apple Immersive Video is an incredible experience, but there will be times when you need to reduce immersion, such as when multitasking, or as part of an interactive application. The portal view is a great reduced-immersion experience, but the default 16:9 aspect ratio may crop some important content. To solve this issue, your playback apps can define a custom wide aspect ratio for the portal viewing mode. Custom aspect ratios can be set when using AVPlayerViewController's AVPlayerViewController.viewport.portal.aspectRatio in AVKit-based apps, or VideoPlayerComponent's VideoPlayerComponent.portalSize in RealityKit-based apps.
Apple Spatial Audio Format
New updates to the Apple Spatial Audio Format (ASAF) Production Suite of AAX plug-ins for Pro Tools give content creators even more creative control. Position objects relative to reference video, and take advantage of significant improvements when using ambisonics, including a new scene compressor plug-in, enhanced heat map drawing, and a refined spatial filtering algorithm. Richer object orientation controls and support for additional channel bed layouts, including standalone LFE and center, expand mixing options. The ASAF Video Player app state now saves with your DAW session, so your workflow picks up right where you left off.
