<!--
{
  "availability" : [
    "visionOS: 27.0.0 -",
    "macOS: 27.0.0 -",
    "iOS: 27.0.0 -",
    "iPadOS: 27.0.0 -",
    "Xcode: 27.0.0 -"
  ],
  "documentType" : "article",
  "framework" : "visionOS",
  "identifier" : "/documentation/visionOS/rendering-gaussian-splats-with-realitykit",
  "metadataVersion" : "0.1.0",
  "role" : "sampleCode",
  "title" : "Rendering Gaussian splats with RealityKit"
}
-->

# Rendering Gaussian splats with RealityKit

Bring a real-world scan into an app by loading splat data from a USD or PLY file.

## Overview

A Gaussian splat asset represents a scene as a large collection of colored, oriented ellipsoids instead of a mesh, so a real-world capture keeps details that are hard to model by hand, like soil texture or the fine edges of leaves.
Each splat carries position, scale, rotation, opacity, and a set of spherical harmonic coefficients that let its color shift depending on the viewing angle.
This reproduces specular highlights and reflections that were present in the original capture.
The capture bakes that color in, so scene lighting doesn’t change how a splat looks, and a splat won’t cast a shadow the way mesh-based content does.

![A Gaussian splat asset, showing a potted succulent sitting on a coffee table next to a chair in a well-lit room.](images/com.apple.visionOS/plant_wwdc_screencapture.png)

This sample uses <doc://com.apple.documentation/documentation/RealityKit> to add a scan of a potted plant to a scene.
The app loads the same plant from either a USD file or a PLY file, and the code that gets that data onto the GPU is shared across every platform the sample runs on.

A scene can only render a limited number of splat entities at once.
When an asset costs more than the renderer can afford, RealityKit reduces its own splat rendering work and reports the change by sending a <doc://com.apple.documentation/documentation/RealityKit/GaussianSplatEvents/RenderingChanged> event on the entity that owns the splat.
The event’s <doc://com.apple.documentation/documentation/RealityKit/GaussianSplatEvents/RenderingChanged/isRenderingLimited> property tells the app whether RealityKit is currently limiting the entity’s splat rendering, and <doc://com.apple.documentation/documentation/RealityKit/GaussianSplatEvents/RenderingChanged/renderingStatus> indicates which rendering quality the system selected.
RealityKit sends the event after it applies that mitigation, so the event reports a decision rather than asking the app to make one.
The sample subscribes to the event so the app’s window can report when RealityKit limits splat rendering.
Reducing the asset’s splat count gives the renderer less to do and brings back full-quality rendering.

> Note: This sample code project is associated with WWDC26 session 287: [Build next generation experiences with visionOS 27](https://developer.apple.com/wwdc26/287), and session 279: [Explore advances in RealityKit](https://developer.apple.com/wwdc26/279).

## Configure the sample code project

Because Gaussian splats don’t render in Simulator, choose a destination from the scheme menu in Xcode and run the app on Apple Vision Pro, iPhone, iPad, or Mac.

The sample shares its loading code for each platform to provide a native environment on launch.
For example, on visionOS, the plant appears in a mixed immersive space where a person can pick it up, move it, and anchor it to a nearby table or floor.
On macOS and iOS it appears in the app’s window, where dragging turns it.

## Load a splat from a USD file

The sample’s bundled USD asset stores its splat data in a primitive whose schema type is `ParticleField3DGaussianSplat`.
This primitive isn’t defined by RealityKit or <doc://com.apple.documentation/documentation/USDKit>, and is instead created by the tool that authored the asset.
`loadUSDEntity(assetName:)` opens the stage with <doc://com.apple.documentation/documentation/USDKit/USDStage/open(_:loadingPayloads:)>, searches the stage’s <doc://com.apple.documentation/documentation/USDKit/USDStage/descendants> for the primitive with that schema using <doc://com.apple.documentation/documentation/USDKit/USDPrim/isSchema(_:)>, and reads the attributes of that <doc://com.apple.documentation/documentation/USDKit/USDPrim> into a `GaussianSplatBuffers`:

```swift
func loadUSDEntity(assetName: String) async throws -> Entity {
    let url = try bundleUSDURL(assetName: assetName)

    let buffers = try await Task.detached(priority: .userInitiated) {
        let stage = try USDStage.open(url)

        guard let splatPrim = stage.descendants.first(where: {
            $0.isSchema("ParticleField3DGaussianSplat")
        }) else {
            throw GaussianSplatError.invalidData("No ParticleField3DGaussianSplat prim found in \(url.lastPathComponent)")
        }

        return try buildBuffers(from: splatPrim)
    }.value

    return try makeSplatEntity(from: buffers, isLinear: true)
}
```

RealityKit expects each rotation as four contiguous floats in `w`, `x`, `y`, `z` order.
<doc://com.apple.documentation/documentation/USDKit/USDValue/Quatf> exposes its parts through the <doc://com.apple.documentation/documentation/USDKit/USDValue/Quatf/real> and <doc://com.apple.documentation/documentation/USDKit/USDValue/Quatf/imaginary> accessors rather than as raw fields in that layout, so the loop reads each component through its accessor and writes it into the matching slot of the <doc://com.apple.documentation/documentation/RealityKit/LowLevelBuffer> instead of copying the structure’s bytes directly:

```swift
private nonisolated func fillRotationBuffer(_ buffer: LowLevelBuffer?, quatArray: [USDValue.Quatf], count: Int) {
    buffer?.withUnsafeMutableBytes { dst in
        let out = dst.bindMemory(to: Float.self)
        for idx in 0..<count {
            let quat = quatArray[idx]
            out[idx * 4 + 0] = quat.real          // w
            out[idx * 4 + 1] = quat.imaginary.x   // x
            out[idx * 4 + 2] = quat.imaginary.y   // y
            out[idx * 4 + 3] = quat.imaginary.z   // z
        }
    }
}
```

For more information on providing raw splat data for rendering Gaussian splats in RealityKit, see <doc://com.apple.documentation/documentation/RealityKit/GaussianSplatComponent>.

## Load a splat from a PLY file

PLY is a generic 3D file format and its header declares the elements and properties inside the file.
PLY doesn’t have a standard schema for Gaussian splat data.
This sample expects a plain-text header that declares each property, followed by a binary block with one record per splat, using the property set that Gaussian splatting tools have settled on as a common convention:

|Property     |Contains                       |Description                                                                                                                                                                                                                                                                                                                             |
|-------------|-------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|`x, y, z`    |3 floats                       |Sets the splat’s center in local space.                                                                                                                                                                                                                                                                                                 |
|`f_dc_0..2`  |3 floats                       |Sets the splat’s base color, which looks the same from every angle.                                                                                                                                                                                                                                                                     |
|`f_rest_0..N`|Varies (0, 9, 24, or 45 floats)|Adds spherical harmonic coefficients that shift the splat’s color based on the viewing angle. The file stores these coefficients as channel-major: all R values, then all G, then all B. The count depends on the <doc://com.apple.documentation/documentation/RealityKit/GaussianSplatResource/SphericalHarmonicDegree> the asset uses.|
|`opacity`    |1 float                        |Sets the splat’s opacity as an unbounded value. RealityKit maps it to 0…1 with the sigmoid function, `1 / (1 + e^-x)`.                                                                                                                                                                                                                  |
|`scale_0..2` |3 floats                       |Sets the splat’s per-axis size, stored in log space. RealityKit converts it with `exp()`.                                                                                                                                                                                                                                               |
|`rot_0..3`   |4 floats                       |Forms a quaternion, in `w`, `x`, `y`, `z` order, that sets the splat’s rotation.                                                                                                                                                                                                                                                        |

`loadPLYEntity(assetName:)` reads the file, deinterleaves those properties into a `GaussianSplatBuffers`, and calls `makeSplatEntity(from:)` without `isLinear`, so RealityKit converts the log-space scale and unbounded opacity:

```swift
func loadPLYEntity(assetName: String) async throws -> Entity {
    let url = try bundlePLYURL(assetName: assetName)
    let buffers = try await Task.detached(priority: .userInitiated) {
        let splatData = try readGaussianSplatFile(url)
        return try deinterleaveGaussianSplatData(splatData)
    }.value
    return try makeSplatEntity(from: buffers)
}
```

The `deinterleaveGaussianSplatData(_:)` method also sanitizes every value it copies, replacing any `NaN` or infinite float with `0`.
RealityKit rejects a splat buffer that contains a non-finite value, so the loader cleans up the capture data before building the buffer.

## Prepare buffers for splat assets

Whichever format the sample starts from, it fills the same GPU buffers for position, scale, rotation, opacity, and spherical harmonics.
This sample defines a custom structure, `GaussianSplatBuffers`, that holds those buffers, and a helper turns a filled-in structure into something the app can add to a scene.
`assembleSplatComponent(from:isLinear:)` packages the buffers into a <doc://com.apple.documentation/documentation/RealityKit/GaussianSplatResource/BufferResource-swift.struct>, describing where each per-splat property lives with a <doc://com.apple.documentation/documentation/RealityKit/GaussianSplatResource/BufferDescriptor>, wraps that in a <doc://com.apple.documentation/documentation/RealityKit/GaussianSplatResource>, and builds a <doc://com.apple.documentation/documentation/RealityKit/GaussianSplatComponent> from it.
`assembleSplatComponent(from:isLinear:)` also sets the resource’s <doc://com.apple.documentation/documentation/RealityKit/GaussianSplatResource/scaleActivation> and <doc://com.apple.documentation/documentation/RealityKit/GaussianSplatResource/opacityActivation>, which tell RealityKit how to interpret the raw numbers in the buffers:

```swift
func assembleSplatComponent(from buffers: GaussianSplatBuffers, isLinear: Bool = false) throws -> GaussianSplatComponent {
    guard let pos = buffers.positionBuffer,
          let scale = buffers.scaleBuffer,
          let rotation = buffers.rotationBuffer,
          let opacity = buffers.opacityBuffer,
          let shBuf = buffers.shBuffer else {
        throw GaussianSplatError.invalidData("One or more GPU buffers failed to allocate")
    }

    let degree = GaussianSplatResource.SphericalHarmonicDegree(rawValue: buffers.degreeSH) ?? .zero
    let bufferResource = try GaussianSplatResource.BufferResource(
        count: Int(buffers.splatCount),
        position: makeDescriptor(buffer: pos, format: .float3, stride: 3 * 4),
        scale: makeDescriptor(buffer: scale, format: .float3, stride: 3 * 4),
        rotation: makeDescriptor(buffer: rotation, format: .float4, stride: 4 * 4),
        opacity: makeDescriptor(buffer: opacity, format: .float, stride: 1 * 4),
        sphericalHarmonics: (makeSHDescriptor(buffer: shBuf, tupleSH: buffers.tupleSH), degree)
    )

    let splatResource = GaussianSplatResource(bufferResource)
    if isLinear {
        splatResource.scaleActivation   = .identity
        splatResource.opacityActivation = .identity
    } else {
        splatResource.scaleActivation   = .exponential
        splatResource.opacityActivation = .sigmoid
    }
    return GaussianSplatComponent(splatResource)
}
```

This sample passes `isLinear: true` for data that’s already in linear space, like the USD version of the plant asset.
For the PLY asset, the sample leaves it `false`, allowing RealityKit to apply the <doc://com.apple.documentation/documentation/RealityKit/GaussianSplatResource/ActivationFunction/exponential> and <doc://com.apple.documentation/documentation/RealityKit/GaussianSplatResource/ActivationFunction/sigmoid> conversions for splat data.

## Add interactivity to a splat entity

A splat entity is an <doc://com.apple.documentation/documentation/RealityKit/Entity> like any other, so it takes the same components.
The sample’s visionOS view adds a <doc://com.apple.documentation/documentation/RealityKit/GroundingShadowComponent> and a <doc://com.apple.documentation/documentation/RealityKit/ManipulationComponent> so a person can pick up the plant and move it.
Because a splat asset has no mesh to derive collision geometry from, the call to <doc://com.apple.documentation/documentation/RealityKit/ManipulationComponent/configureEntity(_:hoverEffect:allowedInputTypes:collisionShapes:)> passes an explicit box from <doc://com.apple.documentation/documentation/RealityKit/ShapeResource/generateBox(size:)>.

```swift
private func configureManipulableObject(_ entity: Entity) {
    ManipulationComponent.configureEntity(
        entity,
        collisionShapes: [ShapeResource.generateBox(size: SIMD3<Float>(repeating: 0.5))]
    )
    var manipulation = entity.components[ManipulationComponent.self] ?? ManipulationComponent()

    // ...

    entity.components.set(manipulation)
    entity.components.set(GroundingShadowComponent(castsShadow: true))
}
```

---

Copyright &copy; 2026 Apple Inc. All rights reserved. | [Terms of Use](https://www.apple.com/legal/internet-services/terms/site.html) | [Privacy Policy](https://www.apple.com/privacy/privacy-policy)