Hi I'm working on a project that uses RealityKit including the placement of 3d objects. However, I want to be able to run the background camera through Metal post-processing before being rendered but haven't been able to find a working approach. I'm open to it rendering directly into the ARview or a separate MTKview or swiftui layer. I've tried using the default xcode project of an Augmented Reality App with Metal Content. However it seems to use a 1.33 aspect camera by default instead of the iphone 15s standard ratio which works by default when I use the regular realitykit pathway and doesnt seem to have the proper ratio available as an option Open to any approach that gets the job done here. Thank you, Any direction would be

How to overlay an image in RealityKit on a 3D model using code so that it does not stretch to the entire object, but has its own height and width that I can change? I have a solution on how to do this, but then it will not be possible to change the height, width or place it anywhere on the 3D model. And this is to cut out a part of the object and overlay the image on the entire cutout area. How to overlay a 2D image on a 3D model without stretching the photo to the entire 3D object? If this is possible, please give an example of how to do this in code. I could not find on the Internet how to do this. Although in other engines this can be done, for example, in Blender or Unity. If I am not mistaken, this is done there using decals

I noticed that with the 4th betas of iOS 18 and visionOS 2, some USDZ models' texture mapping looks completely broken. The issue occurs only with a device, not with the Simulator. It's a regression, the models look fine with iOS 17.5.1 and visionOS 1.2. The issue occurs if I load a model as an Entity in a RealityView iOS or visionOS, or in a SwiftUI 3DModel view on visionOS. Has anyone seen this too? Is there a workaround? I filed a bug report with a minimal example project, it's FB14473756. Screenshot on Vision Pro device: Screenshot on Vision Pro Simulator:

Has anyone gotten EnvironmentLightingConfigurationComponent to work? I tried the code from https://developer.apple.com/documentation/realitykit/environmentlightingconfigurationcomponent to prevent a planet from being lit by the environment. My goal is that the side that isn't lit by the star appears pitch black. However, the code seems to have no effect on visionOS 2 and iPadOS 18 (I tried betas 1 through 4, on device, built with Xcode 16 beta 4). No matter if there is a PointLight or no light at all in the scene, no matter if I use SimpleMaterial or PhysicallyBasedMaterial, no matter if I use a texture or a color on the sphere. I filed a bug report, it's FB14470954. Or am I doing something wrong? Here's my code: var material = PhysicallyBasedMaterial() if let tex = try? await TextureResource(named: "planet.jpg") { material.baseColor = .init(texture: .init(tex)) material.emissiveIntensity = 0 let sphereMesh = MeshResource.generateSphere(radius: 0.5) let entity = ModelEntity() entity.components.set(ModelComponent(mesh: sphereMesh, materials: [material])) entity.position = [-1, 1.0, -1.0] let envLightingConfig = EnvironmentLightingConfigurationComponent(environmentLightingWeight: 0) entity.components.set(envLightingConfig) content.add(entity) }

I am having a difficult time to create particle systems in Reality Composer Pro (visionOS beta 3). They tend to start to flicker and all particles disappear and reappear in semi-random intervals. I can clearly see that happening with one effect that I put inside a small box consisting of 4 transparent walls that has a solid floor. When I change the view angle the particle system starts to flicker when viewed from below its emission height. I tried all combinations of particle rendering: billboard->free, additive etc and it does not change anything. I am using the default particle image. Any help appreciated

Hello, is it possible to take a screenshot of the whole immersive view, including or excluding SwiftUI components? ARView has a snapshot method for this, but it seems there's no equivalent for RealityView. I've tried to use ImageRenderer on a parent of RealityView, but I'm only getting plain white bitmap so far. Thanks in advance, Rlu

I am trying to simulate a pinball game and I want to use PhysicsBody & PhysicsMotion to achieve that. I tuned the parameters around in PhysicsBodyComponent, but the result is not quite ideal for now. Imagine a fully inflated basketball bouncing high off the ground (ground vs basketball). I assign PhysicsBodyComponent and CollisionComponent to both basketball and the ground. For basket ball, I set it as: dynamic mode mass 1, inertia .one Material.Restitution 1 Angular Damping and Linear Damping to 0 AddForce to make the basketball move to hit the ground For ground, I set it as: static mode mass 1, inertia .zero Material.Restitution 1 Angular Damping and Linear Damping to 0 However, when the basket ball hit the ground, it isn't that bouncy, the basketball behaves like hitting to a cotton and the linear speed just dumps fast. Wonder how I could achieve the bouncing effect like real basketball vs ground.

Hi All, I am using RealityKit along with ARKit and Swift UI to develop an app where I am augmenting a usdz model of a complex geometry like that of a car. I have some other usdz files with a simple plane geometry having the material properties embedded within them which also i am loading as model entities. I want to traverse through my car usdz file such that i can pick the material from simple usdz file and apply it to the car as car paint. To do this i know the name of the mesh holding the car paint as well as the name of the material applied. I have tried to traverse through the usdz files using both RealityKit and SceneKit but I am not successful to reach to the lowest mesh and copy the material properties to it. With RealityKit, I have tried to get the instance data using modelEntity as follows :- "sourceModel?.model?.mesh.contents.instances". But this returns instance id, model name and transform only. Any help will be highly appreciated. Thank You

I have a visionOS app that utilizes DrawableQueue and CADisplayLink to update an Entity, TextureResource tied to the drawable, and a Material that uses that TextureResource. TextureResource gets updated with when a video frame is ready. Material properties can get updated from the video or from other sources. Current process: when each video frame is ready, we get the next drawable, render to it, present it, and make an Entity update (e.g. transform). However, I’m experiencing jitter in the rendered content where it seems that the updates to the entity and the drawable being presented are milliseconds off from each other. Should I be using Drawable.presentOnSceneUpdate() to ensure all updates happen in the same update cycle? And if so, do you have any additional details on how to correctly use this function (the docs are unclear)?

Currently, it is not possible to achieve the occlusion effect of the model through depth reading and writing and rendering order, even in the vision 2.0 beta version, this goal cannot be achieved

I have some entities which use attachments to show a label next to them. I would like to change this to only show the label when the entity is being looked at / hovered over. I have the new HoverEffect component on my entity that works nicely, but I can't see how I toggle the visibility of the labels.

I have code such as the following. The performance on the Vision Pro seems to get quite bad once I hit a few thousand of these models. It feels like I should be able to optimise this somehow, perhaps using instancing. Is that possible with RealityKit in visionOS 2? let material = UnlitMaterial(color: .white) let sphereModel = ModelEntity( mesh: .generateSphere(radius: 0.001), materials: [material]) for index in 0..<5000 { let point = generatedPoints[index] let model = sphereModel.clone(recursive: false) model.position = [point.x, point.y, point.z] parent.addChild(starModel) }

我在 window 上层添加scrollerview 并添加 realityview ，实现传送门效果，发现 PortalComponent 效果在某些设备上面无法生效，渲染层级混乱

Hello, The title says it all. I'm not seeing a way to utilize debugOptions when using the SwiftUI RealityView on iOS in the Xcode 16 Beta. Am I missing something? Seems to only be available when using ARView.

Hi all, I'm playing around with RealityKit to see if I can re-use the content for both iOS/macOS and visionOS with the new betas. For the 2D devices, I'm looking at a more traditional, non AR setup. I've fallen over at the first hurdle; dragging an object around on a plane just as a test of how things all work. I was trying to unproject from a plane to the view/window co-ordinates and move the box around based on the result. The code below works if I angle the plane, weirdly; but not if the plane (as I understand it) is 'flat' on the ground. Am I doing this the wrong way? It behaves in a similar fashion with both the default PerspectiveCameraComponent and OrthographicCameraComponent. import SwiftUI import RealityKit struct ContentView: View { var body: some View { RealityView { content in let cubemesh = MeshResource.generateBox(size: 0.2, cornerRadius: 0.05) let cubeModel = ModelEntity(mesh: cubemesh) cubeModel.generateCollisionShapes(recursive: false) cubeModel.components.set(InputTargetComponent()) content.add(cubeModel) let cameraEntity = Entity() cameraEntity.components.set(OrthographicCameraComponent()) //cameraEntity.components.set(PerspectiveCameraComponent()) let cameraPosition: SIMD3<Float> = [10, 10, 5] let target: SIMD3<Float> = .zero cameraEntity.look(at: target, from: cameraPosition, relativeTo: nil) content.add(cameraEntity) } .gesture(DragGesture(coordinateSpace: .global) .targetedToAnyEntity() .onChanged() { value in let planeTransform = Transform(scale: SIMD3<Float>(1, 1, 1), rotation: simd_quatf(angle: 0, axis: SIMD3<Float>(0, 1, 0)), translation: SIMD3<Float>(0, 0, -1)) print(planeTransform.matrix) #if !os(visionOS) if let placementPosition = value.unproject(value.location, from: .global, to: .scene, ontoPlane: (planeTransform.matrix)) { print("projected value:", placementPosition) value.entity.position.x = placementPosition.x value.entity.position.y = placementPosition.y value.entity.position.z = 0 } #endif print(value.location) }) } } #if os(visionOS) #Preview("3D Device", windowStyle: .volumetric) { if #available(visionOS 2.0, *) { ContentView() .volumeBaseplateVisibility(.visible) .frame(depth: 1300) .frame(width: 1280) .frame(height: 1280) } else { ContentView() } } #else #Preview("2D Device") { ContentView() } #endif

If you create a custom shader you get access to a collection of uniform values, one is the uniforms::time() parameter which is defined as "the number of seconds that have elapsed since RealityKit began rendering the current scene" in this doc: https://developer.apple.com/metal/Metal-RealityKit-APIs.pdf Is there some way to get this value from Swift code? I want to animate a value in my shader based on the time so I need to get the starting time value so I can interpolate the animation offset from that point. If I create a System in the update() function I get a SceneUpdateContext instance and that has a deltaTime property but not an elapsedTime property which I would assume would map to the shader time() value.

Hi, I am currently considering porting my AR game from SceneKit to RealityKit so it appears in 3D on Vision OS, but one crucial question to knonw if it can even be ported is: I need a tap on a 3D Model that is than translated into a tap onto the texture of that model. On visionOS this would be the gaze of the person so the question is if I can get the point of a texture is user is looking at (optimally always to I can have a hover effect, but if not at least when tapping the finger together). If that is not possible, is it possible to touch a reality kit object and get the location of that on the texture? All the best Christoph

I'm trying to render a large number of entities, it looks like each ModelEntity causes a draw call, even if you share the ModelComponent so each Entity shares the mesh and materials. I tried to use the MeshInstanceCollection inside MeshResource to generate a large number of objects in the scene, the code works and draws many objects but the draw count is still one call per instance, this seems strange I would assume it should only be one draw call for the single entity since I have specified to use instancing in the resource. Has anybody else successfully used instancing in RealityKit to draw a large number go Entities (maybe around 10,000) or drawn this amount of items successfully with 60fps any other way? Here is some sample code that draws 100 cubes using instancing but still causes 100 draw calls. func instanceTest(scene: RealityKit.Scene) { var resource = MeshResource.generateBox(size: 0.2) var contents = MeshResource.Contents() contents.models = resource.contents.models var arr: [MeshResource.Instance] = [] var matrix = matrix_identity_float4x4 matrix[3, 0] = 0.5 for i in 0..<100 { let inst = MeshResource.Instance(id: "\(i)", model: "MeshModel", at: matrix) arr.append(inst) } contents.instances = MeshInstanceCollection(arr) let updatedResource = try? MeshResource.generate(from: contents) let unlitMaterial = UnlitMaterial(color: .red) let modelEntity = ModelEntity( mesh: updatedResource!, materials: [unlitMaterial] ) let anchor = AnchorEntity() anchor.addChild(modelEntity) scene.addAnchor(anchor) }

I'd like to create meshes in RealityKit ( AR mode on iPad ) in screen-space, i.e. for UI. I noticed a lot of useful new functionality in RealityKit for the next OS versions, including the OrthographicCameraComponent here: https://developer.apple.com/documentation/realitykit/orthographiccameracomponent?changes=_3 I think this would help, but I need AR worldtracking as well as a regular perspective camera to work with the 3D elements. Firstly, can I have a camera attached selectively to a few entities, just for those entities? This could be the orthographic camera. Secondly, can I make it so those entities are always rendered in-front, in screenspace? (They'd need to follow the camera.) If I can't have multiple cameras, what can be done in that case? Is it actually better to use a completely different view / API for layering on-top of RealityKit? I would much rather keep everything in RealityKit, however, for simplicity.

I am trying to establish a workflow with using Reality Composer Pro to make scenes - I am grey boxing a scene using primitives at the moment. I have set up a cube with a texture material and a simple animation to spin. I am confused as to what I should be loading. I have created what I think is a scene asset in the package for the Reality Composer Project. Here is a code snippet: struct ContentView: View { var body: some View { RealityView { content in do { let scene = try await ModelEntity(named: "HOF") content.add(scene) } catch { print("Error loading scene: \(error.localizedDescription)") } } } } Here is the project layout in Reality Composer Pro: