Reality Kit 3 large scale practices - what is under the hood

Question

just_some_game_dev OP

Created 3w

Replies 1

Boosts 1

Participants 2

Hello guys I want to use Reality Kit 3 for none AR games.

Coming from unreal (7y) I have a few technical questions if i may please:

(all questions are assuming my game will handle 50-500 enemies on screen)

How does RTK3 handle everything under the hood? its easy to create a struct that hold multiple floats as a data blocks for the system to handle (like hp, dmg, ect), but, what happen when i want generic ASSET data (materials, meshes) ?

What if i want to construct a material and hand a cheap id to the processor (system) ?

while make sure i packed it nicely to the cpu's cache line .

There is not much in the documentation to explain the under the hood architecture of the system for me to make educated decision abut my code. i try to avoid reference bloat in my game, and make sure the system does not coupled new material or new mesh every time it want to perform a generic command on the entity (in 60 fps)

For example, in unreal mass, as bloated and over complicated it is - its pretty clear to me what is construction phase, what is the recommended way to generate my data blocks (called fragments there) and how exactly to use them in the system so its all tightly packed

At this moment my only assumption is to create a 3 layers custom system to make sure its all running properly:

The builder of the pool - it will do the " let shinyMetal = SimpleMaterial(color: ... ect" and save it on some dictionary map
the data block - just a simple struct that hold a thin reference to the pool that can be use as a component in the ECS
the system - The RTK3 built is system real time . can use the struct as a component and even swap the id in real time

is this safe consider how RTK3 build under the hood? or is this even slower ? i have very little to go by. if reality kit engineer can reply and help me here it would be awesome :)

Thank you so much for reading guys ! was kinda long. cheers

Answered by DTS Engineer in 896065022

Thanks for the detailed write-up.

Your three-layer instinct maps cleanly onto RealityKit's documented Entity Component System (ECS):

Your data blocks (hp, dmg) are custom components: value-type structs conforming to Component. Attach them to entities and keep per-entity state there.
Your system layer is a class conforming to System, with init(scene:) and update(context:). RealityKit calls update(context:) once per scene update. Inside it, you fetch just the entities you care about with an EntityQuery such as EntityQuery(where: .has(EnemyComponent.self)), then iterate them with context.entities(matching:updatingSystemWhen:). That is the documented pattern for behavior across many entities, rather than a per-entity update call. Register the system once with registerSystem(), and order systems with dependencies when needed.

For the "construct a material, hand a cheap handle, avoid reference bloat" concern: RealityKit provides that handle. MeshResource is a reference type, and ModelComponent holds a MeshResource and an array of Material. Create your mesh and material once and reuse the same instances across every entity's ModelComponent, rather than constructing new ones per entity. You do not need a separate ID-to-asset pool, because the resource reference is that shared handle.

For drawing many copies of the same mesh at scale, RealityKit provides MeshInstancesComponent, which performs GPU instancing. It is available on iOS, iPadOS, macOS, tvOS, and visionOS 26 and later. You attach it to one entity and supply per-instance transforms through LowLevelInstanceData, and RealityKit renders all the copies from that single model. If your enemies share geometry, that is the mechanism for drawing many of them from one model, and worth measuring against separate entities for your workload. One boundary worth being clear about: those instanced copies render as sub-entities of that one entity, not independent entities. Per-enemy state and behavior stay on real entities in the ECS: hp, damage, AI, and collisions.

On what you can depend on: the dependable contract is the ECS API above and the resource model. RealityKit does not publish its internal component storage or memory layout, so I can't confirm details such as cache-line packing. I would not design hard dependencies around an assumed layout, since anything unpublished can change between releases.

For performance at your scale, RealityKit publishes guidance aimed at exactly this. Reaching 60 fps means staying within about 16.6 ms of main-thread time per frame. The documented levers most relevant to a large entity count are to keep each update(context:) doing as little as possible and to flatten assets so fewer entities carry per-frame ECS work. RealityKit also issues a draw call per mesh, so merging meshes that share a material reduces render-thread cost. Apply those, measure on device, and if it still falls short for your case, that is what a Feedback report is for: include your test project and measurements so the team that owns this area can see the scenario. The same applies if the documentation is too thin for the data-oriented details you need.

Please let me know if there are any questions.

Answer 1

DTS Engineer OP

Apple

2w

Accepted Answer

Recommended

Thanks for the detailed write-up.

Your three-layer instinct maps cleanly onto RealityKit's documented Entity Component System (ECS):

Your data blocks (hp, dmg) are custom components: value-type structs conforming to Component. Attach them to entities and keep per-entity state there.
Your system layer is a class conforming to System, with init(scene:) and update(context:). RealityKit calls update(context:) once per scene update. Inside it, you fetch just the entities you care about with an EntityQuery such as EntityQuery(where: .has(EnemyComponent.self)), then iterate them with context.entities(matching:updatingSystemWhen:). That is the documented pattern for behavior across many entities, rather than a per-entity update call. Register the system once with registerSystem(), and order systems with dependencies when needed.

For the "construct a material, hand a cheap handle, avoid reference bloat" concern: RealityKit provides that handle. MeshResource is a reference type, and ModelComponent holds a MeshResource and an array of Material. Create your mesh and material once and reuse the same instances across every entity's ModelComponent, rather than constructing new ones per entity. You do not need a separate ID-to-asset pool, because the resource reference is that shared handle.

For drawing many copies of the same mesh at scale, RealityKit provides MeshInstancesComponent, which performs GPU instancing. It is available on iOS, iPadOS, macOS, tvOS, and visionOS 26 and later. You attach it to one entity and supply per-instance transforms through LowLevelInstanceData, and RealityKit renders all the copies from that single model. If your enemies share geometry, that is the mechanism for drawing many of them from one model, and worth measuring against separate entities for your workload. One boundary worth being clear about: those instanced copies render as sub-entities of that one entity, not independent entities. Per-enemy state and behavior stay on real entities in the ECS: hp, damage, AI, and collisions.

On what you can depend on: the dependable contract is the ECS API above and the resource model. RealityKit does not publish its internal component storage or memory layout, so I can't confirm details such as cache-line packing. I would not design hard dependencies around an assumed layout, since anything unpublished can change between releases.

For performance at your scale, RealityKit publishes guidance aimed at exactly this. Reaching 60 fps means staying within about 16.6 ms of main-thread time per frame. The documented levers most relevant to a large entity count are to keep each update(context:) doing as little as possible and to flatten assets so fewer entities carry per-frame ECS work. RealityKit also issues a draw call per mesh, so merging meshes that share a material reduces render-thread cost. Apply those, measure on device, and if it still falls short for your case, that is what a Feedback report is for: include your test project and measurements so the team that owns this area can see the scenario. The same applies if the documentation is too thin for the data-oriented details you need.

Please let me know if there are any questions.