I currently have an iOS app that transmits h264 code through wifi, uses videotoolbox to decode and displays it with MTKView, and I want to implement similar functions in visionOS. What should I do? MTKView is not available on visionOS

Hi, I'm trying to have an entity (and some attachments to it) to rotate. If I add the entity to content, add the attachment as a child entity, and set the entity as InputTargetComponent, then when I add a gesture ONLY the entity rotates and NOT the attachments (added as child entities). If I add a parent entity with let parentEntity = ModelEntity(), add my entity to the parentEntity, then add the attachments to an entity (which is now a child of the ModelEntity) and set the ModelEntity as InputTargetComponent then the whole thing rotates (including attachments) I'm sure there must be a bug, why would it work only with an added ModelEntity? Anyway, bug or not a bug, the problem I have now is that it rotates around the axes of the ModelEntity, not my primary entity, which is what I want. Is there a way to set the ModelEntity axes to be the axes of my primary child entity so it rotates like I want? What call should I use to move the axes where would I find the axes of the first child entity which should be the focus of my app? Here is my code: var body: some View { RealityView { content, attachments in // Add the initial RealityKit content if let specimenentity = try? await Entity(named: "Immersive", in: realityKitContentBundle) { let parentEntity = ModelEntity() parentEntity.addChild(specimenentity) content.add(parentEntity) let entityBounds = specimenentity.visualBounds(relativeTo: parentEntity) parentEntity.collision = CollisionComponent(shapes: [ShapeResource.generateBox(size: entityBounds.extents).offsetBy(translation: entityBounds.center)]) parentEntity.generateCollisionShapes (recursive: true) parentEntity.components.set(InputTargetComponent()) if let Left_Hemisphere = attachments.entity(for: "Left_Hemisphere") { //4. Position the Attachment and add it to the RealityViewContent Left_Hemisphere.position = [-0.5, 1, 0] specimenentity.addChild(Left_Hemisphere) } } } attachments: { Attachment(id: "Left_Hemisphere") { //2. Define the SwiftUI View Text("Left_Hemisphere") .font(.extraLargeTitle) .padding() .glassBackgroundEffect() } } .gesture( DragGesture() .targetedToAnyEntity() .onChanged { value in let entity = value.entity var orientation = Rotation3D(entity.orientation(relativeTo: nil)) var newOrientation: Rotation3D if (value.location.x >= lastGestureValueX) { newOrientation = orientation.rotated(by: .init(angle: .degrees(0.5), axis: .y)) } else { newOrientation = orientation.rotated(by: .init(angle: .degrees(-0.5), axis: .y)) } entity.setOrientation(.init(newOrientation), relativeTo: nil) lastGestureValueX = value.location.x orientation = Rotation3D(entity.orientation(relativeTo: nil)) if (value.location.y >= lastGestureValueY) { newOrientation = orientation.rotated(by: .init(angle: .degrees(0.5), axis: .x)) } else { newOrientation = orientation.rotated(by: .init(angle: .degrees(-0.5), axis: .x)) } entity.setOrientation(.init(newOrientation), relativeTo: nil) lastGestureValueY = value.location.y } ) } }

I'm following the Meet Reality Composer Pro walkthrough and ran into something that didn't function as expected. When I got to the step where I add five "Bird_With_Audio.usda" references to the scene, I found they did not play audio. After some trial and error, I found that Preview > Resource in each of their Spatial Audio items was set to "None." If I click the dropdown menu, I see several "Bird_Calls" groups to pick from. I checked the original Bird_With_Audio.usda that I had created, and the "Bird_Calls" audio group was correctly assigned and worked. I tried dragging a sixth Bird_With_Audio into the scene and confirmed that the Spatial Audio item suddenly empties, rendering the bird silent. I was able to go through each of the five birds and set their Spatial Audio Resource to Bird_Calls, and the group worked like the video demonstrates. While this fixed the issue, as a beginner I'd like to know why this happened. It doesn't seem right that I would build and item and then have to re-attach any sounds to it when I place it in the main scene. So…where did I mess up?

I'm trying to make a simple demo of using ShaderGraphMaterial in a USDZ file that I can preview on Mac and VisionOS but I'm having trouble. In Reality Composer, I make a sphere, then assign a ShaderGraphMaterial to the material, with a simple diffuse color (green) input. When I save the file as .usda, it displays as a gray sphere on mac rather than the green sphere shown in reality composer. If I then convert to usdz using Reality Converter, I get a warning on import: "Shader nodes must have “id” as the implementationSource, with id values that begin with “Usd”. Also, shader inputs with connections must each have a single, valid connection source." And the exported .usdz also shows as a gray sphere. Is there a simple demo of a .usda file using ShaderGraphMaterial that displays on Mac, iOS, and VisionOS that I can look at to see how it looks internally? My actual problem is creating usdz / usda files on visionOS for viewing on iOS / Mac / VisionOS.. but the first step is showing it's possible to even use ShaderGraphMaterial across all platforms. Thanks

I would like to add text to a Reality Composer Pro scene and set the actual text via code. How can I achieve this? I haven't seen any "Text" element in the editor.

I've got a couple 2D PNG assets that I want to add to a scene made of a couple other udsz files in RCP (picture adding a couple 2D videogame characters to a simple 3D diorama). When I try to drag the PNGs to the workspace or the file tree…nothing happens. I found a walkthrough on Medium (called "Importing and Exporting Personalized Objects for Augmented Reality: Reality Composer and SwiftUI" for those curious as I can't link to Medium posts here) that makes it look like users could do this with simple drag-and-drop. The Medium post is from June 2023, and in the screenshots RCP visually looks a lot more like Reality Composer on iPad, so I'm assuming it's changed a lot since then? Is there still a way to do this? I've tried adding the 2D elements to a scene with Blenders "import images as planes," but I'm getting weird halos around them and was hoping RCP could make the process a bit easier/cleaner.

We are porting a iOS Unity AR app to native visionOS. Ideally, we want to re-use our AR models in both applications. These AR models are rather simple. But still, converting them manually would be time-consuming, especially when it gets to the shaders. Is anyone aware of any attempts to write conversion tools for this? Maybe in other ecosystems like Godot or Unreal, where folks also want to convert the proprietary Unity format to something else? I've seen there's an FBX converter, but this would not care for shaders or particles. I am basically looking for something like the Polyspatial-internal conversion tools, but without the heavy weight of all the rest of Unity. Alternatively, is there a way to export a Unity project to visionOS and then just take the models out of the Xcode project?

I have a custom material in Reality Composer. When I attach it to a cube and try loading the scene in XCode, the material cannot be cast to a ShaderGraphMaterial because it has been changed to a PhysicallyBasedMaterial. The material was always a Custom material, I did not change the type in Reality Composer. Does anyone know how to fix?

Hi, I create an entity and add a bunch of attachments (code is based on the Diorama demo). I can rotate the entity with this: .gesture( DragGesture() .targetedToAnyEntity() .onChanged { value in let entity = value.entity let orientation = Rotation3D(entity.orientation(relativeTo: nil)) let newOrientation: Rotation3D if (value.location.x >= lastGestureValue) { newOrientation = orientation.rotated(by: .init(angle: .degrees(0.5), axis: .y)) } else { newOrientation = orientation.rotated(by: .init(angle: .degrees(-0.5), axis: .y)) } entity.setOrientation(.init(newOrientation), relativeTo: nil) lastGestureValue = value.location.x } ) But the attachments stay still. How can I rotate the entity AND the attachment at the same time?

I'm following the Meet Reality Composer Pro walkthrough and ran into something that didn't function as expected. When I got to the step where I add five "Bird_With_Audio.usda" references to the scene, I found they did not play audio. After some trial and error, I found that Preview > Resource in each of their Spatial Audio items was set to "None." If I click the dropdown menu, I see several "Bird_Calls" groups to pick from. I checked the original Bird_With_Audio.usda that I had created, and the "Bird_Calls" audio group was correctly assigned and worked. I tried dragging a sixth Bird_With_Audio into the scene and confirmed that the Spatial Audio item suddenly empties, rendering the bird silent. I was able to go through each of the five birds and set their Spatial Audio Resource to Bird_Calls, and the group worked like the video demonstrates. While this fixed the issue, as a beginner I'd like to know why this happened. It doesn't seem right that I would build and item and then have to re-attach any sounds to it when I place it in the main scene. So…where did I mess up?

Is it possible to use an image sequence, .mov or sprite sheet as a node source for a custom material in Reality Composer Pro? I have noticed that in the particle emitter, the magic preset uses a 4x4 sprite sheet as a particle source. Can this be done within the shader graph for the diffuse or normal slot?

I am struggling to figure out how to make a shader to animate each vertex of a model separately using noise. I watched a video on how to do this in Unity, but I think something must be different with how Reality Composer Pro handles the noise nodes? For example, in this graph I just hooked up the noise node directly to the geometry modifier: In my output you can see the plane is adjust per-vertex using the noise node. My goal would be to animate this like waves, but moving the noise. So in this graph I use time with sin to adjust the UV of the noise. This seems to change the noise node to output a single value (I guess that makes sense, since I modify the UV, it results in a single value, at that UV in the noise map). So then, I take that as the Y value and put it back into the geometry modifier. But now it doesn't work per-vertex, it moves the whole model up and down (based on the single value coming out of the noise map). How do I make this apply to each vertex of the noise map individually? This is an example of the output I want in Unity, the plane is being adjusted per-vertex by a scrolling 2d noise node:

I have been digging into learning shader graphs by watching Unity shader graph content, cause lots of the same concepts apply. One thing I noticed was that in Unity, each node in the shader graph has a little preview. I don't think this exists in Reality Composer Pro, but is there anyway to mimic it (like can I hook up a node that allows me to debug the graph at that point?) If not, I'm happy to just file a feedback about it, but just thought I'd ask!

We are building an AR experience for deployment on iphones. We are using Unity but it looks as if Reality Composer Pro has better features for spatial audio. I am not sure if Reality Composer Pro can only be used for Vision Pro or can it also be used for deployment on Iphone or ipad.

Hi guys, if you started using Vision Pro, I'm sure you already found some limitations. Let's join forces and make feature requests. When creating Feedback, request from one guy may not get any attenption from Apple, but if we join and more of us make the same request, we might just push those ideas through. Feel free to add your ideas and don't forget to create feedback: app windows can only be moved forward to a distance of about 20ft/6m. I'm pretty sure some users would like to push window as far as a few miles away and make the window large to be still legible. This would be very interesting especialy when using Environments and 360-degree view. I really want to put some apps up on the sky above the mountains and around me, even those iOS apps just made compatible with Vision Pro. when capturing screen, I always get message "Video capture not possible due to insufficient lighting". Why? I have Environment loaded and extended 360 degrees with some apps opened, so there is no need for external lighting (at least I thing it's not needed). I just want to capture what I see. Imagine creating tutorials, recording lessons for learning various subjects, etc. Actual Vision Pro user might prefer loading their on environments an setup app in spatial domain, but for those that don't have it yet or when creating videos to be available on antique 2D computer screens , it may be useful to create 2D videos this way. 3D video recording is not very good, kind of shaky, not when Vision Pro is static, but when walking and especially when turning head left/right/up/down (even relatively slowly). I think hardware should be able to capture and create nice and smooth video. It's possible that Apple just designed simple camera app and wants to give developers a chance to create a better Camera app, but it still would be nice to have something better out of the box. I would like to be able to walk through Environments. I understand safety of see-through effect, so users didn't hit any obstacles, but perhaps obstacles could be detected and when user gets to 6ft/2m from obstacle then it could present at first warning (there is already "You are close to and object" and then make surroundigns visible, but if there are no obstacles (user can be located in large space and can place a tape or a thread around the safe area), I should be able to walk around and take a look inside that crater on the Moon. We need Environments, Environments, Environments and yet more of them, I was hoping for hundreds, so we could even pick some of them and use in our apps, like games where you want to setup a specific environment. Well, that's just a beginning and I could go on and on and on, but tell me what you guys think. Regards and enjoy new virtual adventure! Robert

I am very new to shaders, never used one of the large systems like Unity. However I have started exploring visionOS programming and that led me to create some effects for materials in Reality Composer Pro. I have been overwhelmed with the possibilities, but also kind of lost. I understand that RCPs shaders are based on MaterialX, so maybe there are tutorials on the web that would cover how to create procedural effects (fire, wind, water, etc)? I’ve stumbled through…but it’s slow going. Are there any good resources that talk about how to use the various nodes to create procedural effects? For example, it took me a while to figure out that using the “time” node allows me to animate cool color changes, especially when combined with various math and remap nodes. Just looking for some basic resources I think. Would the shader graph tutorials about Unity, apply to using RCP? Are the node types similar enough?

Hi folks! I have been working with a team on a Vision Pro app using Reality Composer Pro. One thing we have found is that multiple developers editing the RCPro scene are a continuous problem, similar to when multiple developers edit a storyboard. RC Pro maintains a SceneMetadataList.json file that indexes the file contents of the project that is updated even as the scene hierarchy is opened and closed, not to mention other changes to scene content. We are getting frequent continuous version control conflicts with this file as we each make changes and edits to the scene, or even browse the scene without making any substantive changes. It seems like it would be safe to add the SceneMetadataList.json file in a RC Pro project to .gitignore. Is that recommended? Any downsides to that?

Following this thread I'm able to render a simple picture in a Plane material, however, I'm unable to scale it to show bigger than the window itself, or move it behind the window. Here's my relevant code so far.- var body: some View { ZStack { RealityView { content in var material = UnlitMaterial() material.color = try! .init(tint: .white, texture: .init(.load(named: "image", in: nil))) let entity = Entity() let component = ModelComponent( mesh: .generatePlane(width: 1, height: 1), materials: [material] ) entity.components.set(component) let currentTransform = entity.transform var newTransform = Transform(scale: currentTransform.scale, rotation: currentTransform.rotation, translation: SIMD3(0, 0, -0.2)) entity.move(to: newTransform, relativeTo: nil) /* let scalingPivot = Entity() scalingPivot.position.y = entity.visualBounds(relativeTo: nil).center.y scalingPivot.addChild(entity) content.add(scalingPivot) scalingPivot.scale *= .init(x: 1, y: 1, z: 1) */ } } } It belongs to an ImmersiveSpace I'm opening directly from my main window, but I have several issues: The texture shows always in front of the window I'm unable to scale it (scaling seems to affect to the texture coordinates inside the material instead of scaling the mesh itself) I can only see the texture in the canvas preview (not in simulator)

I'm developing a vision pro application. However, when the user takes off the Apple Vision Pro device, the application goes into the background. How can I prevent this behavior programmatically?

I'd like to map a SwiftUI view (in my case: a map) onto a 3D curved plane in immersive view, so user can literally immersive themselves into the map. The user should also be able to interact with the map, by panning it around and selecting markers. Is this possible?