Hello, I was wondering how I can initialize an ImageAnchoringSource using
https://developer.apple.com/documentation/realitykit/anchoringcomponent/imageanchoringsource/init(_:)
When I construct one using a URL, it doesn't seem to be tracked and I see in the following when I debug print the component:
▿ 0 : AnchoringComponent
▿ target : Target
▿ referenceImage : 1 element
▿ from : ImageAnchoringSource
▿ url : Optional<URL>
▿ some : file:///var/mobile/Containers/Data/Application/D1126EA0-A1D7-468F-A40C-8578B7F5BDDF/Library/Caches/CodeCache/0E457AA7-2195-48B9-9DD4-58CEB9397F69.png
- _url : file:///var/mobile/Containers/Data/Application/D1126EA0-A1D7-468F-A40C-8578B7F5BDDF/Library/Caches/CodeCache/0E457AA7-2195-48B9-9DD4-58CEB9397F69.png
- _parseInfo : nil
- _baseParseInfo : nil
- name : nil
- group : nil
▿ trackingMode : TrackingMode
- trackingMode : 2
Is there a specific format for the parseInfo?
When I use the same image to make an image anchoring source by group and name in AR Resources, it is tracked.
Thank you!
RealityKit
RSS for tagSimulate and render 3D content for use in your augmented reality apps using RealityKit.
Posts under RealityKit tag
200 Posts
Selecting any option will automatically load the page
Post
Replies
Boosts
Views
Activity
I'm converting my game from SceneKit to RealityKit. It has a SpriteKit overlay that according to Explore advanced rendering with RealityKit 2 I can add with the code below.
The code runs fine if the SKScene only contains a SKSpriteNode (see the commented out line), but when I add a SKShapeNode with a fillColor instead, the app crashes with this error:
-[MTLDebugRenderCommandEncoder validateCommonDrawErrors:]:5970: failed assertion `Draw Errors Validation
MTLDepthStencilDescriptor uses frontFaceStencil but MTLRenderPassDescriptor has a nil stencilAttachment texture
MTLDepthStencilDescriptor uses backFaceStencil but MTLRenderPassDescriptor has a nil stencilAttachment texture
'
I don't know enough about low-level graphics and stencils yet to figure out a quick solution, so I would appreciate if anyone could share an easy fix or explanation of what's wrong. Thanks!
class ViewController: NSViewController {
var device: MTLDevice!
var renderer: SKRenderer!
override func loadView() {
let arView = ARView(frame: NSScreen.main!.frame)
view = arView
arView.renderCallbacks.prepareWithDevice = { [weak self] device in
guard let self = self else { return }
self.device = device
renderer = SKRenderer(device: MTLCreateSystemDefaultDevice()!)
let scene = SKScene()
let shape = SKShapeNode(rectOf: CGSize(width: 10, height: 10))
shape.fillColor = .red
scene.addChild(shape)
// scene.addChild(SKSpriteNode(color: .red, size: CGSize(width: 10, height: 10)))
renderer.scene = scene
}
arView.renderCallbacks.postProcess = { [weak self] context in
guard let self = self else { return }
let encoder = context.commandBuffer.makeBlitCommandEncoder()
encoder?.copy(from: context.sourceColorTexture, to: context.targetColorTexture)
encoder?.endEncoding()
renderer.update(atTime: context.time)
let descriptor = MTLRenderPassDescriptor()
descriptor.colorAttachments[0].loadAction = .load
descriptor.colorAttachments[0].storeAction = .store
descriptor.colorAttachments[0].texture = context.targetColorTexture
renderer.render(withViewport: CGRect(x: 0, y: 0, width: context.targetColorTexture.width, height: context.targetColorTexture.height), commandBuffer: context.commandBuffer, renderPassDescriptor: descriptor)
}
}
}
I'm in the process of converting my SceneKit game to RealityKit. In SceneKit I used to be able to mark nodes as selected by setting SCNMaterial.emission with a custom color. I can do the same with PhysicallyBasedMaterial.emissiveColor, but I'd like to keep my entitities unaffected by the scene lights by using UnlitMaterial. In SceneKit I can set a category mask to indicate what light should affect what node, but there doesn't seem to be such a thing in RealityKit. So at the moment it seems like I have to choose between being able to mark an entity as selected, or having entities unaffected by lighting, but not both.
Is there some effect or component I can use to mark entities as selected by applying some coloring regardless of the material used?
I want to fade objects in and out, and while setting an entity's OpacityComponent works, animating it doesn't seem to do anything.
In the following code the second sphere should fade out, but it keeps its initial opacity. On the other hand, the animation that changes its transform works. What am I doing wrong?
class ViewController: NSViewController {
override func loadView() {
let arView = ARView(frame: NSScreen.main!.frame)
let anchor = AnchorEntity(.world(transform: matrix_identity_float4x4))
arView.scene.addAnchor(anchor)
let sphere = ModelEntity(mesh: .generateSphere(radius: 0.5))
anchor.addChild(sphere)
sphere.components.set(OpacityComponent(opacity: 0.1))
let sphere2 = ModelEntity(mesh: .generateSphere(radius: 0.5))
sphere2.position = .init(x: 0.2, y: 0, z: 0)
anchor.addChild(sphere2)
sphere2.components.set(OpacityComponent(opacity: 0.1))
sphere.playAnimation(try! AnimationResource.makeActionAnimation(for: FromToByAction(to: 0, timing: .linear), duration: 1, bindTarget: .opacity))
sphere.playAnimation(try! AnimationResource.makeActionAnimation(for: FromToByAction(to: Transform(translation: SIMD3(x: 0.1, y: 0, z: 0)), timing: .linear), duration: 1, bindTarget: .transform))
view = arView
}
}
I'm trying to convert my game from SceneKit to RealityKit. I noticed that even when the scene is static (nothing moves), RealityKit keeps using CPU. In SceneKit, CPU goes down to 0% with a static scene.
With this simplest of games, RealityKit keeps using about 65% CPU:
class ViewController: NSViewController {
override func loadView() {
view = ARView(frame: NSScreen.main!.frame)
}
}
Is this expected or a bug?
I created FB22125047.
I have an iOS app that uses RealityView to display some models and interact with them, and the app uses regular iOS app navigations, then a challenge I'm facing is how to maintain multiple RealityView on multiplescreens.
For example Screen A has a RealityView, and then I navigate to Screen B (also has a RealityView) using stack based navigation, when I do so I got a crash
-[MTLDebugRenderCommandEncoder validateCommonDrawErrors:]:5970: failed assertion `Draw Errors Validation
Fragment Function(fsRealityPbr): argument envProbeTable[0] from Buffer(7) with offset(0) and length(16) has space for 16 bytes, but argument has a length(864).
Fragment Function(fsRealityPbr): incorrect type of texture (MTLTextureType2D) bound at Texture binding at index 20 (expect MTLTextureTypeCubeArray) for envProbeDiffuseArray[0].
Interestingly this crash only happens when debugging with Xcode, not happens when the app runs on its own.
I'm not sure what I'm doing is anti-pattern or it's some Xcode debugging limitation.
Hi,
Is there a resource or sample code about how to draw an outline around a mesh in RealityKit?
Typically, this is useful for visualizing a selection, like in Reality Composer Pro.
How to achieve such effect? A shader material? A post-process effect in ARView or RealityRenderer?
Methods such as duplicating the entity mesh, scaling it, and using material.faceCulling = .front did not look good in my experiments.
Thank you.
I'm currently converting my game from SceneKit to RealityKit. What's the easiest way to run an animation in which every frame I want to run custom code, similar to SCNAction.customAction which accepts a callback that is called repeatedly until the action completes?
Hi, I’m wondering whether RealityKit has its own scene management system, since it uses ARView (backed by ARKit) to present AR content. Does RealityKit manage scenes independently, or does it rely entirely on ARKit’s scene handling?
Thank you.
My visionOS 26.3 app displays a diorama-like scene in a RealityView in a mixed immersive space, about 1 meter square, with view attachments floating above the scene.
Each view attachment fades out after user interaction, by animating the view's opacity.
What I'm observing is that depending on the position of a view attachment relative to the scene and the camera, an unwanted cutout effect is observed (presumably because of draw order issues), as shown in the right column in the screenshots below.
YouTube video link of these sequences: https://youtu.be/oTuo0okKCkc
(19 seconds)
My question:
How does visionOS determine the view attachment draw order relative to the RealityView scene?
If I better understood how the draw order is determined, I could modify my scene to ensure that the view attachments were always drawn after the scene, fixing the unwanted cutout effect.
I've successfully used ModelSortGroupComponent to control the draw order of entities within the RealityView scene, but my understanding is that this approach cannot be used with view attachments.
I've submitted FB22014370 about this issue.
Thank you.
I have a simple visionOS app that creates an Entity, writes it to the device, and then attempts to load it. However, when the entity file get overwritten, it affects the ability for the app to load it correctly.
Here is my code for saving the entity.
import SwiftUI
import RealityKit
import UniformTypeIdentifiers
struct ContentView: View {
var body: some View {
VStack {
ToggleImmersiveSpaceButton()
Button("Save Entity") {
Task {
// if let entity = await buildEntityHierarchy(from: urdfPath) {
let type = UTType.realityFile
let filename = "testing.\(type.preferredFilenameExtension ?? "bin")"
let documentsURL = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask)[0]
let fileURL = documentsURL.appendingPathComponent(filename)
do {
let mesh = MeshResource.generateBox(size: 1, cornerRadius: 0.05)
let material = SimpleMaterial(color: .blue, isMetallic: true)
let modelComponent = ModelComponent(mesh: mesh, materials: [material])
let entity = Entity()
entity.components.set(modelComponent)
print("Writing \(fileURL)")
try await entity.write(to: fileURL)
} catch {
print("Failed writing")
}
}
}
}
.padding()
}
}
Every time I press "Save Entity", I see a warning similar to:
Writing file:///var/mobile/Containers/Data/Application/1140E7D6-D365-48A4-8BED-17BEA34E3F1E/Documents/testing.reality
Failed to set dependencies on asset 1941054755064863441 because NetworkAssetManager does not have an asset entity for that id.
When I open the immersive space, I attempt to load the same file:
import SwiftUI
import RealityKit
import UniformTypeIdentifiers
struct ImmersiveView: View {
@Environment(AppModel.self) private var appModel
var body: some View {
RealityView { content in
guard
let type = UTType.realityFile.preferredFilenameExtension
else {
return
}
let documentsURL = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask)[0]
let fileURL = documentsURL.appendingPathComponent("testing.\(type)")
guard FileManager.default.fileExists(atPath: fileURL.path) else {
print("❌ File does not exist at path: \(fileURL.path)")
return
}
if let entity = try? await Entity(contentsOf: fileURL) {
content.add(entity)
}
}
}
}
I also get errors after I overwrite the entity (by pressing "Save Entity" after I have successfully loaded it once). The warnings that appear when the Immersive space attempts to load the new entity are:
Asset 13277375032756336327 Mesh (RealityFileAsset)URL/file:///var/mobile/Containers/Data/Application/1140E7D6-D365-48A4-8BED-17BEA34E3F1E/Documents/testing.reality/Mesh_0.compiledmesh failure: Asset provider load failed: type 'RealityFileAsset' -- RERealityArchive: Failed to open load stream for entry 'assets/Mesh_0.compiledmesh'.
Asset 8308977590385781534 Scene (RealityFileAsset)URL/file:///var/mobile/Containers/Data/Application/1140E7D6-D365-48A4-8BED-17BEA34E3F1E/Documents/testing.reality/Scene_0.compiledscene failure: Asset provider load failed: type 'RealityFileAsset' -- RERealityArchive: Failed to read archive entry.
AssetLoadRequest failed because asset failed to load '13277375032756336327 Mesh (RealityFileAsset)URL/file:///var/mobile/Containers/Data/Application/1140E7D6-D365-48A4-8BED-17BEA34E3F1E/Documents/testing.reality/Mesh_0.compiledmesh' (Asset provider load failed: type 'RealityFileAsset' -- RERealityArchive: Failed to open load stream for entry 'assets/Mesh_0.compiledmesh'.)
The order of operations to make this happen:
Launch app
Press "Save Entity" to save the entity
"Open Immersive Space" to view entity
Press "Save Entity" to overwrite the entity
"Open Immersive Space" to view entity, failed asset load request
Also
Launch app, the entity should still be save from last time the app ran
"Open Immersive Space" to view entity
Press "Save Entity" to overwrite the entity
"Open Immersive Space" to view entity, failed asset load request
NOTE: It appears I can get it to work slightly better by pressing the "Save Entity" button twice before attempting to view it again in the immersive space.
When using the new RealityKit Manipulation Component on Entities, indirect input will never translate the entity - no matter what settings are applied. Direct manipulation works as expected for both translation and rotation.
Is this intended behaviour? This is different from how indirect manipulation works on Model3D. How else can we get translation from this component?
visionOS 26 Beta 2
Build from macOS 26 Beta 2 and Xcode 26 Beta 2
Attached is replicable sample code, I have tried this in other projects with the same results.
var body: some View {
RealityView { content in
// Add the initial RealityKit content
if let immersiveContentEntity = try? await Entity(named: "MovieFilmReel", in: reelRCPBundle) {
ManipulationComponent.configureEntity(immersiveContentEntity, allowedInputTypes: .all, collisionShapes: [ShapeResource.generateBox(width: 0.2, height: 0.2, depth: 0.2)])
immersiveContentEntity.position.y = 1
immersiveContentEntity.position.z = -0.5
var mc = ManipulationComponent()
mc.releaseBehavior = .stay
immersiveContentEntity.components.set(mc)
content.add(immersiveContentEntity)
}
}
}
My application calculates three distinct Meesus Double [x, y, z] Radian values to light a sphere in RealityKit with DirectionalLight. It is my understanding that I must use (simd_quatf) for each radian value to properly light the sphere in the view. The code correctly [orientates] the sphere with the combined (simd_quatf) DirectionalLight in the view, but the illumination (Z-axis) fails to properly illuminate the sphere with the expected result, compared to associated Meesus web page images. For the moment, I do not know how to correct the (Z-axis). Curious for a suggestion ... :]
// Location values.
let theLatitude: Double = 51.13107260
let theLongitude: Double = -114.01127910
let currentDate: Date = Date()
struct TheCalculatedMoonPhaseTest_ContentView: View {
var body: some View {
VStack {
if #available(macOS 15.0, *) {
RealityView { content in
let moonSphere_Entity = Entity.createSphere(radius: 0.90, color: .black)
moonSphere.Entity.name = "MoonSphere"
moonSphere.Entity.position = SIMD3<Float>(x: 0, y: 0, z: 0)
content.add(moonSphere.Entity)
let sunLight_Entity = createDirectionalLight(latitude: theLatitude, longitude: theLongitude, date: currentDate)
content.add(sunLight_Entity)
} // End of [RealityView]
} else {
// Earlier version required.
} // End of [if #available(macOS 15.0, *)]
} // End of [VStack]
.background(Color.black)
} // End of [var body: some View]
// MARK: - 🟠🟠🟠🟠 [SET THE BACKGROUND COLOUR] 🟠🟠🟠🟠
var backgroundColor: Color = Color.init(.black)
// MARK: - 🟠🟠🟠🟠 [CREATE THE DIRECTIONAL LIGHT FOR THE SPHERE] 🟠🟠🟠🟠
func createDirectionalLight(latitude: Double, longitude: Double, date: Date) -> Entity {
let directionalLight = DirectionalLight()
directionalLight.light.color = .white
directionalLight.light.intensity = 1000000
directionalLight.shadow = DirectionalLightComponent.Shadow()
directionalLight.shadow?.maximumDistance = 5
directionalLight.shadow?.depthBias = 1
// MARK: 🟠🟠🟠🟠 Retrieve the [MEESUS MOON AGE VALUES] from the [CONSTANT FOLDER] 🟠🟠🟠🟠
let theMeesusMoonAge_LunarAgeDaysValue = 25.90567592898601
if theMeesusMoonAge_LunarAgeDaysValue >= 23.10 && theMeesusMoonAge_LunarAgeDaysValue < (29.530588853 - 1.00) {
let someCalculatedX_WestEastRadian: Float = Float(1.00)
// Identify the sphere’s DirectionalLight Tilt Angle (Y) radian value ::
// Note :: The following Tilt Angle is corrected to [Zenith] with the [MeesusCalculatedTilt_Angle] minus the [MeesusCalculatedPar_Angle].
let someCalculatedY_TiltAngleRadian: Float = Float(1.3396086)
// Identify the sphere’s DirectionalLight Illumination (Z) radian Value ::
// Note :: The Meesus calculated illumination fraction is converted to degrees, then converted to a radian value.
let someCalculatedZ_IlluminationAngleRadian: Float = Float(0.45176168630244457) // <=== 14.3800% Illumination.
// Define rotation angles in radians for X, Y, and Z axes.
let x_Radians = someCalculatedX_WestEastRadian
let y_Radians = someCalculatedY_TiltAngleRadian
let z_Radians = someCalculatedZ_IlluminationAngleRadian
// Identify and separate the quaternion [simd_quatf] for each Radian.
let q_X = simd_quatf(angle: x_Radians, axis: SIMD3<Float>(1, 0, 0))
let q_Y = simd_quatf(angle: y_Radians, axis: SIMD3<Float>(0, 1, 0))
let q_Z = simd_quatf(angle: z_Radians, axis: SIMD3<Float>(0, 0, 1))
// Apply and combine the rotations, where order matters.
let combinedRotation = q_Z * q_Y * q_X
// Identify the [Combined Rotation].
// The [MyMoonMeesus] :: [WANING CRESCENT] calculated [combinedRotation] :: simd_quatf(real: 0.73715997, imag: SIMD3<Float>(0.24427173, 0.61516714, -0.13599981)) ° Radians
// Normalize the [combinedRotation].
let theNormalizesRotation = simd_normalize(combinedRotation)
// Identify the [Normalized Combined Rotation].
// The [MyMoonMeesus] :: [WANING CRESCENT] calculated [normalizedRotation] :: simd_quatf(real: 0.73715997, imag: SIMD3<Float>(0.24427173, 0.61516714, -0.13599981)) ° Radians
// Assume the [theNormalizesRotation] appears reversed.
let theCorrectedRotation = theNormalizesRotation.inverse
// Identify the [Reversed Combined Rotation].
// The [MyMoonMeesus] :: [WANING CRESCENT] calculated [correctedRotation] :: simd_quatf(real: 0.73715997, imag: SIMD3<Float>(-0.24427173, -0.61516714, 0.13599981)) ° Radians
// Apply the [Corrected Rotation] to the entity.
directionalLight.transform.rotation *= theCorrectedRotation
// Add the [directionalLight] to the scene ::
let anchor = AnchorEntity()
anchor.addChild(directionalLight)
} // End of [if theMeesusMoonAge_LunarAgeDaysValue >= 23.10 && theMeesusMoonAge_LunarAgeDaysValue < (29.530588853 - 1.00)]
return directionalLight
} // End of [func createDirectionalLight(latitude: Double, longitude: Double, date: Date) -> Entity]
} // End of [struct TheCalculatedMoonPhaseTest_ContentView: View]
// MARK: 🟠🟠🟠🟠 [ENTITY HELPER EXTENSION] 🟠🟠🟠🟠
extension Entity {
static func createSphere(radius: Float, color: NSColor) -> Entity {
let mesh = MeshResource.generateSphere(radius: radius)
var material = PhysicallyBasedMaterial()
material.baseColor = .init(tint: color)
let modelComponent = ModelComponent(mesh: mesh, materials: [material])
let entity = Entity()
entity.components.set(modelComponent)
entity.components.set(Transform())
return entity
} // End of [static func createSphere(radius: Float, color: NSColor) -> Entity]
} // End of [extension Entity]
// Application Image :: Calgary
// Website Image :: timeanddate
// mooncalc.org
Hi, I'm trying to use AnchorEntity for horizontal surfaces.
It works when the entity is being created, but I'm looking for a way to snap this entity to the nearest surface, after translating it, for example with a DragGesture.
What would be the best way to achieve this? Using raycast, creating a new anchor, trackingMode to continuous etc.
Do I need to use ARKitSession as I want continuous tracking?
Hi, is there a way to track feet in a visionOS app in an immersive space? I want the whole body to be visible in VR, and I want to know if the user touches an object with their foot.
Hello, I'm struggling trying to interact with a RealityKit entity nested (or at least visually nested) in a second one.
I think I've the same issue as mentioned on this StackOverflow post, but I can't manage to reproduce the solution. https://stackoverflow.com/questions/79244424/how-to-prioritize-a-specific-entity-when-collision-boxes-overlap-in-realitykit
What I'd like to achieve is to translate the red box using a DragGesture, while still be able to interact using a TapGesture on the sphere. Currently, I can only do one at a time.
Does anyone know the solution?
extension CollisionGroup {
static let parent: CollisionGroup = CollisionGroup(rawValue: 1 << 0)
static let child: CollisionGroup = CollisionGroup(rawValue: 1 << 1)
}
struct ImmersiveView: View {
var body: some View {
RealityView { content in
let boxMesh = MeshResource.generateBox(size: 0.35)
let boxMaterial = SimpleMaterial(color: .red.withAlphaComponent(0.25), isMetallic: false)
let boxEntity = ModelEntity(mesh: boxMesh, materials: [boxMaterial])
let sphereMesh = MeshResource.generateSphere(radius: 0.05)
let sphereMaterial = SimpleMaterial(color: .blue, isMetallic: false)
let sphereEntity = ModelEntity(mesh: sphereMesh, materials: [sphereMaterial])
content.add(sphereEntity)
content.add(boxEntity)
boxEntity.components.set(InputTargetComponent())
boxEntity.components.set(
CollisionComponent(
shapes: [ShapeResource.generateBox(size: SIMD3<Float>(repeating: 0.35))],
isStatic: true,
filter: CollisionFilter(
group: .parent,
mask: .parent.subtracting(.child)
)
)
)
sphereEntity.components.set(InputTargetComponent())
sphereEntity.components.set(HoverEffectComponent())
sphereEntity.components.set(
CollisionComponent(
shapes: [ShapeResource.generateSphere(radius: 0.05)],
isStatic: true,
filter: CollisionFilter(
group: .child,
mask: .child.subtracting(.parent)
)
)
)
}
}
}
VideoMaterial Black Screen on Vision Pro Device (Works in Simulator)
App Overview
App Name: Extn Browser
Bundle ID: ai.extn.browser
Purpose: A visionOS web browser that plays 360°/180° VR videos in an immersive sphere environment
Development Environment & SDK Versions
Component
Version
Xcode
26.2
Swift
6.2
visionOS Deployment Target
26.2
Swift Concurrency
MainActor isolation enabled
App is released in the TestFlight.
Frameworks Used
SwiftUI - UI framework
RealityKit - 3D rendering, MeshResource, ModelEntity, VideoMaterial
AVFoundation - AVPlayer, AVAudioSession
WebKit - WKWebView for browser functionality
Network - NWListener for local proxy server
Sphere Video Mechanism
The app creates an immersive 360° video experience using the following approach:
// 1. Create sphere mesh (10 meter radius for immersive viewing)
let mesh = MeshResource.generateSphere(radius: 10.0)
// 2. Create initial transparent material
var material = UnlitMaterial()
material.color = .init(tint: .clear)
// 3. Create entity and invert sphere (negative X scale)
let sphere = ModelEntity(mesh: mesh, materials: [material])
sphere.scale = SIMD3<Float>(-1, 1, 1) // Inverts normals for inside-out viewing
sphere.position = SIMD3<Float>(0, 1.5, 0) // Eye level
// 4. Create AVPlayer with video URL
let player = AVPlayer(url: videoURL)
// 5. Configure audio session for visionOS
let audioSession = AVAudioSession.sharedInstance()
try audioSession.setCategory(.playback, mode: .moviePlayback, options: [.mixWithOthers])
try audioSession.setActive(true)
// 6. Create VideoMaterial and apply to sphere
let videoMaterial = VideoMaterial(avPlayer: player)
if var modelComponent = sphere.components[ModelComponent.self] {
modelComponent.materials = [videoMaterial]
sphere.components.set(modelComponent)
}
// 7. Start playback
player.play()
ImmersiveSpace Configuration
// browserApp.swift
ImmersiveSpace(id: appModel.immersiveSpaceID) {
ImmersiveView()
.environment(appModel)
}
.immersionStyle(selection: .constant(.mixed), in: .mixed)
Entitlements
<!-- browser.entitlements -->
<key>com.apple.security.app-sandbox</key>
<true/>
<key>com.apple.security.network.client</key>
<true/>
<key>com.apple.security.network.server</key>
<true/>
Info.plist Network Configuration
<key>NSAppTransportSecurity</key>
<dict>
<key>NSAllowsArbitraryLoads</key>
<true/>
</dict>
The Issue
Behavior in Simulator: Video plays correctly on the inverted sphere surface - 360° video is visible and wraps around the user as expected.
Behavior on Physical Vision Pro: The sphere displays a black screen. No video content is visible, though the sphere entity itself is present.
Important: Not a DRM/Licensing Issue
This issue is NOT related to Digital Rights Management (DRM) or FairPlay. I have tested with:
Unlicensed raw MP4 video files (no DRM protection)
Self-hosted video content with no copy protection
Direct MP4 URLs from CDN without any licensing requirements
The same black screen behavior occurs with all unprotected video sources, ruling out DRM as the cause.
(Plain H.264 MP4, no DRM)
Screen Recording: Working in Simulator
The following screen recording demonstrates playing a 360° YouTube video in the immersive sphere on the visionOS Simulator:
https://cdn.commenda.kr/screen-001.mov
This confirms that the VideoMaterial and sphere rendering work correctly in the simulator, but the same setup shows a black screen on the physical Vision Pro device.
Observations
AVPlayer status reports .readyToPlay - The video appears to load successfully
VideoMaterial is created without errors - No exceptions thrown
Sphere entity renders - The geometry is visible (black surface)
Audio session is configured - No errors during audio session setup
Network requests succeed - The video URL is accessible from the device
Same result with local/unprotected content - DRM is not a factor
Console Logs (Device)
The logging shows:
Sphere created and added to scene
AVPlayer created with correct URL
VideoMaterial created and applied
Player status transitions to .readyToPlay
player.play() called successfully
Rate shows 1.0 (playing)
Despite all success indicators, the rendered output is black.
Questions for Apple
Are there known differences in VideoMaterial behavior between the visionOS Simulator and physical Vision Pro hardware?
Does VideoMaterial(avPlayer:) require specific video codec/format requirements that differ on device? (The test video is a standard H.264 MP4)
Is there a required Metal capability or GPU feature for VideoMaterial that may not be available in certain contexts on device?
Does the immersion style (.mixed) affect VideoMaterial rendering on hardware?
Are there additional entitlements required for video texture rendering in RealityKit on physical hardware?
Attempted Solutions
Configured AVAudioSession with .playback category
Added delay before player.play() to ensure material is applied
Verified sphere scale inversion (-1, 1, 1)
Tested multiple video URLs (including raw, unlicensed MP4 files)
Confirmed network connectivity on device
Ruled out DRM/FairPlay issues by testing unprotected content
Environment Details
Device: Apple Vision Pro
visionOS Version: 26.2
Xcode Version: 26.2
macOS Version: Darwin 25.2.0
Recently, applications for the Swift Student Challenge opened up. I noticed, that when selected "where to run your app" (mine was developed in Xcode 26), and you select Xcode26, there is a note underneath it that basically says all Xcode projects will be run on the simulator.
What if my project is dependent on AR? How would I let the judges test my submission?
Topic:
Community
SubTopic:
Swift Student Challenge
Tags:
Swift Student Challenge
ARKit
AR / VR
RealityKit
Hello,
I am experiencing an issue where lighting is disabled within a Portal Component specifically when using Progressive or Full Immersion styles in visionOS 26.2.
[Steps to Reproduce]
Create a scene in Reality Composer Pro with custom lighting (Directional Light, IBL, etc.) and load it as an Entity.
Add a PortalComponent to this Entity.
Observe the Entity in an ImmersiveSpace under different Immersion Styles.
[Observed Behavior]
Mixed Immersion: Lighting works as expected.Progressive / Full Immersion: Lighting is disabled, causing the content to render incorrectly.
[Additional Context]
Without PortalComponent: The same Reality Composer Pro scene renders lighting correctly across all Immersion Styles (Mixed, Progressive, and Full).Regression: In applications built with Xcode 16.4 / visionOS 2.5, lighting inside the Portal functioned correctly. This issue appears to have emerged with Xcode 26.2 and visionOS 26.2.
[Questions]
Is the disabling of lighting inside Portals during Full/Progressive Immersion an intended architectural change or optimization in visionOS 26.2?
Are there any workarounds, such as specific PortalComponent configurations or new API flags, to re-enable lighting in these immersion modes?
Thank you.
Topic:
Spatial Computing
SubTopic:
Reality Composer Pro
Tags:
AR / VR
RealityKit
Reality Composer Pro
My procedural animation using IKRig occasionally jitters and throws this warning. It completely breaks immersion.
getMoCapTask: Rig must be of type kCoreIKSolverTypePoseRetarget
Any idea where to start looking or what could be causing this?