I am creating a RealityKit scene that will contain over 12,000 duplicate cubes arranged in a circle (see image below). This is for some high-energy physical simulations I am doing. I accomplish this scene by creating a single cube and cloning it a bunch of times. So, I there is a single MeshResource and Material even though there are a lot of entities. I have confirmed this by checking with Swift's === operator. Even with this, the program is unworkably slow.
Any suggestions or tricks that could help with this type of scene?
Using a single geometry was the trick to getting SceneKit to work fast with geometries like this. I've been updating my software to RealityKit because I far prefer the structure of RealityKit over SceneKit.
RealityKit
RSS for tagSimulate and render 3D content for use in your augmented reality apps using RealityKit.
Post
Replies
Boosts
Views
Activity
On Scenekit, using SCNShapewe can create SCN geometry from SwiftUI 2D shapes/beziers:https://developer.apple.com/documentation/scenekit/scnshape
Is there an equivalent in RealityKit?
Could we use the generate(from:) for that?https://developer.apple.com/documentation/realitykit/meshresource/3768520-generate
https://developer.apple.com/documentation/realitykit/meshresource/3768520-generate
Hello Dev Community,
I've been thinking over Apple's preference for USDZ for AR and 3D content, especially when there's the widely used GLTF. I'm keen to discuss and hear your insights on this choice.
USDZ, backed by Apple, has seen a surge in the AR community. It boasts advantages like compactness, animation support, and ARKit compatibility. In contrast, GLTF too is a popular format with its own merits, like being an open standard and offering flexibility.
Here are some of my questions toward the use of USDZ:
Why did Apple choose USDZ over other 3D file formats like GLTF?
What benefits does USDZ bring to Apple's AR and 3D content ecosystem?
Are there any limitations of USDZ compared to other file formats?
Could factors like compatibility, security, or integration ease have influenced Apple's decision?
I would love to hear your thoughts on this. Feel free to share any experiences with USDZ or other 3D file formats within Apple's ecosystem!
Hi,
is there a way in visionOS to anchor an entity to the POV via RealityKit?
I need an entity which is always fixed to the 'camera'.
I'm aware that this is discouraged from a design perspective as it can be visually distracting. In my case though I want to use it to attach a fixed collider entity, so that the camera can collide with objects in the scene.
Edit:
ARView on iOS has a lot of very useful helper properties and functions like cameraTransform (https://developer.apple.com/documentation/realitykit/arview/cameratransform)
How would I get this information on visionOS? RealityViews content does not seem offer anything comparable.
An example use case would be that I would like to add an entity to the scene at my users eye-level, basically depending on their height.
I found https://developer.apple.com/documentation/realitykit/realityrenderer which has an activeCamera property but so far it's unclear to me in which context RealityRenderer is used and how I could access it.
Appreciate any hints, thanks!
I have a RealityView in my visionOS app. I can't figure out how to access RealityRenderer. According to the documentation (https://developer.apple.com/documentation/realitykit/realityrenderer) it is available on visionOS, but I can't figure out how to access it for my RealityView. It is probably something obvious, but after reading through the documentation for RealityView, Entities, and Components, I can't find it.
has anyone gotten their 3d Models to render in seperate windows, i tried following the code in the video for creating a seperate window group, but i get a ton of obsecure errors, i was able to get it to render in my 2d windows, but when i try making a seperate window group i get errors
I have some strange behavior in my app. When I set the position to .zero you can see the sphere normally. But when I change it to any number it doesn't matter which and how small. The Sphere isn't visible or in the view.
The RealityView
import SwiftUI
import RealityKit
import RealityKitContent
struct TheSphereOfDoomRV: View {
@StateObject var viewModel: SphereViewModel = SphereViewModel()
let sphere = SphereEntity(radius: 0.25, materials: [SimpleMaterial(color: .red, isMetallic: true)], name: "TheSphere")
var body: some View {
RealityView { content, attachments in
content.add(sphere)
} update: { content, attachments in
sphere.scale = SIMD3<Float>(x: viewModel.scale, y: viewModel.scale, z: viewModel.scale)
} attachments: {
VStack {
Text("The Sphere of Doom is one of the most powerful Objects. You can interact with him in every way you can imagine ").multilineTextAlignment(.center)
Button {
} label: {
Text("Play Video!")
}
}.tag("description")
}.modifier(GestureModifier()).environmentObject(viewModel)
}
}
SphereEntity:
import Foundation
import RealityKit
import RealityKitContent
class SphereEntity: Entity {
private let sphere: ModelEntity
@MainActor
required init() {
sphere = ModelEntity()
super.init()
}
init(radius: Float, materials: [Material], name: String) {
sphere = ModelEntity(mesh: .generateSphere(radius: radius), materials: materials)
sphere.generateCollisionShapes(recursive: false)
sphere.components.set(InputTargetComponent())
sphere.components.set(HoverEffectComponent())
sphere.components.set(CollisionComponent(shapes: [.generateSphere(radius: radius)]))
sphere.name = name
super.init()
self.addChild(sphere)
self.position = .zero // .init(x: Float, y: Float, z: Float) and [Float, Float, Float] doesn't work ...
}
}
So if I drag an entity in RealityView I have to disable the PhysicsBodyComponent to make sure nothing fights dragging the entity around. This makes sense.
When I finish a drag, this closure gets executed:
.gesture(
DragGesture()
.targetedToAnyEntity()
.onChanged { e in
// ...
}
.onEnded { e in
let velocity: CGSize = e.gestureValue.velocity
}
If I now re-add PhysicsBodyComponent to the component I just dragged, and I make it mode: .dynamic it will loose all velocity and drop straight down through gravity.
Instead the solution is to apply mode: .kinematic and also apply a PhysicsMotionComponent component to the entity. This should retain velocity after letting go of the object.
However, I need to instatiate it with PhysicsMotionComponent(linearVelocity: SIMD3<Float>, angularVelocity: SIMD3<Float>).
How can I calculate the linearVelocity and angularVelocity when the e.gestureValue.velocity I get is just a CGSize?
Is there another prop of gestureValue I should be looking at?
I'm developing 3D Scanner works on iPad.
I'm using AVCapturePhoto and Photogrammetry Session.
photoCaptureDelegate is like below:
extension PhotoCaptureDelegate: AVCapturePhotoCaptureDelegate {
func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto photo: AVCapturePhoto, error: Error?) {
let fileUrl = CameraViewModel.instance.imageDir!.appendingPathComponent("\(PhotoCaptureDelegate.name)\(id).heic")
let img = CIImage(cvPixelBuffer: photo.pixelBuffer!, options: [ .auxiliaryDepth: true, .properties: photo.metadata ])
let depthData = photo.depthData!.converting(toDepthDataType: kCVPixelFormatType_DepthFloat32)
let colorSpace = CGColorSpace(name: CGColorSpace.sRGB)
let fileData = CIContext().heifRepresentation(of: img, format: .RGBA8, colorSpace: colorSpace!, options: [ .avDepthData: depthData ])
try? fileData!.write(to: fileUrl, options: .atomic)
}
}
But, Photogrammetry session spits warning messages:
Sample 0 missing LiDAR point cloud!
Sample 1 missing LiDAR point cloud!
Sample 2 missing LiDAR point cloud!
Sample 3 missing LiDAR point cloud!
Sample 4 missing LiDAR point cloud!
Sample 5 missing LiDAR point cloud!
Sample 6 missing LiDAR point cloud!
Sample 7 missing LiDAR point cloud!
Sample 8 missing LiDAR point cloud!
Sample 9 missing LiDAR point cloud!
Sample 10 missing LiDAR point cloud!
The session creates a usdz 3d model but scale is not correct.
I think the point cloud can help Photogrammetry session to find right scale, but I don't know how to attach point cloud.
I know that CustomMaterial in RealityKit can update texture by use DrawableQueue, but in new VisionOS, CustomMaterial doesn't work anymore. How i can do the same thing,does ShaderGraphMaterial can do?I can't find example about how to do that. Looking forward your repley, thank you!
In my project, i want to use new shadergraphmaterial to do the stereoscopic render, i notice that there is a node called Camera Index Switch Node can do this. But when i tried it , i found that :
It can only output Integer type value, when i change to float value , it change back again, i don't konw if it is a bug.
2. So i test this node with a IF node,i found that it output is weird.
Below is zero should output,it is black
but when i change to IF node,it is grey,it is neither 0 nor 1(My IF node result is TRUE result 1, FALSE result 0)
I wanna ask if this is a bug, and if this is a correct way to do the stereoscopic render.
Is it possible to animate some property on a RealityKit component? For example, the OpacityComponent has an opacity property that allows the opacity of the entities it's attached to, to be modified. I would like to animate the property so the entity fades in and out.
I've been looking at the animation API for RealityKit and it either assumes the animation is coming from a USDZ (which this is not), or it allows properties of entities themselves to be animated using a BindTarget. I'm not sure how either can be adapted to modify component properties?
Am I missing something?
Thanks
Hello all,
I am building for visionOS with another engineer and using Reality Composer Pro to validate usd files.
The starting position of my animated usdz, its position when it's first loaded, is not the same as the first frame of the animation on the usdz file
For testing, I am using the AR Quick Look asset 'toy_biplane_idle.usdz' which demonstrates the same 'error' we're currently getting with our own usdz files.
When the usdz is loaded, it is on the ground plane -
But when the aniamtion is played, the plane 'snaps' to the position of the first frame of the animation -
This 'snapping' behavior is giving us problems. We want the user ot see this plane in its static 'load' position with the option to play the animation. But we dont want it to snap when the user presses play
Is it possible to load the .usdz in the position specified by the first frame of the animation? What is the best way to fix this issue.
Thanks!
In RealityKit in visionOS 1.0 I'm perplexed that PhysicallyBasedMaterial and CustomMaterial have faceCulling properties but ShaderGraphMaterial does not.
Is there some way to achieve front face culling in a shader graph without creating a separate mesh with reversed triangle vertex indices?
Im trying to programmatically alter a a video frame before applying it to geometry using VideoMaterial. What I'm finding is that the output appears as though no videoCompositor was applied to the playerItem. Is this expected behavior? Is there a work around besides using an ExportSession to bounce the movie to disk?
Context being VisionOS development, I was trying to do something like
let root = ModelEntity()
child1 = ModelEntity(...)
root.addChild(child1)
child2 = ModelEntity(...)
root.addChild(child2)
only to find that, despite seemingly being together, I can only pick by children entities when I apply a DragGesture in VisionOS. Any idea what's going on?
I have tried entity.generateCollisionShapes (generating simple box-shaped collision) and ShapeResource.generateConvex(from: entity) (generating convex-shaped collision, as the name suggests). Unfortunately neither suits my case, where I have a torus entity inside which no collision should happen with other entities - namely, smaller entities should be able to "fall through" the torus, thus the title of this post.
Was wondering if there's any solution that I overlooked. Thanks 🙏
In a RealityView, I have scene loaded from Reality Composer Pro. The entity I'm interacting with has a PhysicallyBasedMaterial with a diffuse color. I want to change that color when on long press. I can get the entity and even get a reference to the material, but I can't seem to change anything about it.
What is the best way to change the color of a material at runtime?
var longPress: some Gesture {
LongPressGesture(minimumDuration: 0.5)
.targetedToAnyEntity()
.onEnded { value in
value.entity.position.y = value.entity.position.y + 0.01
if var shadow = value.entity.components[GroundingShadowComponent.self] {
shadow.castsShadow = true
value.entity.components.set(shadow)
}
if let model = value.entity.components[ModelComponent.self] {
print("material", model)
if let mat = model.materials.first {
print("material", mat)
// I have a material here but I can't set any properties?
// mat.diffuseColor does not exist
}
}
}
}
Here is the full code
struct Lab5026: View {
var body: some View {
RealityView { content in
if let root = try? await Entity(named: "GestureLab", in: realityKitContentBundle) {
root.position = [0, -0.45, 0]
if let subject = root.findEntity(named: "Cube") {
subject.components.set(HoverEffectComponent())
subject.components.set(GroundingShadowComponent(castsShadow: false))
}
content.add(root)
}
}
.gesture(longPress.sequenced(before: dragGesture))
}
var longPress: some Gesture {
LongPressGesture(minimumDuration: 0.5)
.targetedToAnyEntity()
.onEnded { value in
value.entity.position.y = value.entity.position.y + 0.01
if var shadow = value.entity.components[GroundingShadowComponent.self] {
shadow.castsShadow = true
value.entity.components.set(shadow)
}
if let model = value.entity.components[ModelComponent.self] {
print("material", model)
if let mat = model.materials.first {
print("material", mat)
// I have a material here but I can't set any properties?
// mat.diffuseColor does not exist
// PhysicallyBasedMaterial
}
}
}
}
var dragGesture: some Gesture {
DragGesture()
.targetedToAnyEntity()
.onChanged { value in
let newPostion = value.convert(value.location3D, from: .global, to: value.entity.parent!)
let limit: Float = 0.175
value.entity.position.x = min(max(newPostion.x, -limit), limit)
value.entity.position.z = min(max(newPostion.z, -limit), limit)
}
.onEnded { value in
value.entity.position.y = value.entity.position.y - 0.01
if var shadow = value.entity.components[GroundingShadowComponent.self] {
shadow.castsShadow = false
value.entity.components.set(shadow)
}
}
}
}
Is there a way to give a "Primitive Shape" entity created through Reality Composer Pro a ModelComponent?
I have a custom ShaderGraphMaterial assigned to a primitive shape in my RC Pro scene hierarchy, and I'd like to tweak the inputs of this material programatically. I found a great example of the behavior I'm looking for here: https://developer.apple.com/videos/play/wwdc2023/10273/?time=1862
@State private var sliderValue: Float = 0.0
Slider(value: $sliderValue, in: (0.0)...(1.0))
.onChange(of: sliderValue) { _, _ in
guard let terrain = rootEntity.findEntity(named: "DioramaTerrain"),
var modelComponent = terrain.components[ModelComponent.self],
var shaderGraphMaterial = modelComponent.materials.first
as? ShaderGraphMaterial else { return }
do {
try shaderGraphMaterial.setParameter(name: "Progress", value: .float(sliderValue))
modelComponent.materials = [shaderGraphMaterial]
terrain.components.set(modelComponent)
} catch { }
}
}
However, when I try applying this example to my use-case, my project's equivalent to this line fails to execute:
var modelComponent = terrain.components[ModelComponent.self]
The only difference I can see between my case and this example is my entity is a primitive shape, whereas the example uses a model reference to a .usdz file. Is there some way to update a primitive shape entity to contain this ModelComponent in its set of components so I can reference + update its materials programmatically?
Really new to the Vision OS coding. We tried to create a customized experience just like what apple did in the system Environments or the Disney did in its app ( it seems they put user in a carefully made 3d model). Do we need to make a 3d model for the environment and how to put it around the user sphere like the skybox?