State-of-the-Art 3D (no AR) on macOS using RealityKit?

What is the current recommendation for creating high-quality 3D content? The context is a hobbyist, specialised CAD app for macOS (with an iPadOS companion) that is mostly 2D but also offers a 3D visualization option (currently OpenGL).

Somewhere down the line there might be an AR view but at the moment - certainly for macOS - it's purely generated 3D visualization, all rendered content.

So starting with a rewrite of the 3D visualization in 2024 targeting macOS Sequoia/iPadOS 18 is RealityKit the suggested way forward?

Cheers,
Jay

Answered by Vision Pro Engineer in 790327022

Answers inline.

What is the current recommendation for creating high-quality 3D content?

Use the 3D modeling tool of your choice to create models. When you are happy with your model convert it to USD (.usd, .usda, .usdc, .usdz) so it can be used with RealityKit and Reality Composer Pro. USDZ Tools (available for download here is one way to do this.

So starting with a rewrite of the 3D visualization in 2024 targeting macOS Sequoia/iPadOS 18 is RealityKit the suggested way forward?

If RealityKit meets your needs and you don't have an immediate need to support older iOS versions then it's a good choice (over SceneKit). Here's some useful information to help you make that determination:

As you and Heckj mentioned, Apple announced cross-platform APIs for RealityKit in iOS Beta 18. Watch Discover RealityKit APIs for iOS, macOS and visionOS to learn more about it. If you haven't already, I encourage you to watch the 2023 and 2024 sessions on RealityKit. If you are new to RealityKit start with Build spatial experiences with RealityKit.

Finally familiarize yourself with Reality Composer Pro. Use it to easily compose, edit, and preview 3D content. Meet Reality Composer Pro and Compose interactive 3D content in Reality Composer Pro are 2 good sessions to start with.

SceneKit hasn't been deprecated, so I'm sure the official response would be along the lines of "it's up to you", but RealityKit is (to my watching) where the updates are coming in and what's getting used. RealityView is now also available on macOS (although I haven't downloaded the new beta to try it out) - setting up a RealityKit view on macOS was previously a bit of challenge. That said, this year brings even more options for programmatic (or procedural) generation of meshes and textures that should make visualizations as you describe far more accessible.

Look into LowLevelMesh for the raw mesh generation types. There's equivalent bits for textures.

Answers inline.

What is the current recommendation for creating high-quality 3D content?

Use the 3D modeling tool of your choice to create models. When you are happy with your model convert it to USD (.usd, .usda, .usdc, .usdz) so it can be used with RealityKit and Reality Composer Pro. USDZ Tools (available for download here is one way to do this.

So starting with a rewrite of the 3D visualization in 2024 targeting macOS Sequoia/iPadOS 18 is RealityKit the suggested way forward?

If RealityKit meets your needs and you don't have an immediate need to support older iOS versions then it's a good choice (over SceneKit). Here's some useful information to help you make that determination:

As you and Heckj mentioned, Apple announced cross-platform APIs for RealityKit in iOS Beta 18. Watch Discover RealityKit APIs for iOS, macOS and visionOS to learn more about it. If you haven't already, I encourage you to watch the 2023 and 2024 sessions on RealityKit. If you are new to RealityKit start with Build spatial experiences with RealityKit.

Finally familiarize yourself with Reality Composer Pro. Use it to easily compose, edit, and preview 3D content. Meet Reality Composer Pro and Compose interactive 3D content in Reality Composer Pro are 2 good sessions to start with.

I think of mesh generation as Model I/O territory. But I always use it with SceneKit. They probably are thrown in together. Although Heckj's mention of "LowLevelMesh" makes me want to look at RealityKit again.

Accepted Answer

JayMcBee here's an answer to your followup question:

Yes. Consider this example that creates a metallic sphere and adds it to the RealityView's content:

let mesh = MeshResource.generateSphere(radius: 0.2)
let material = SimpleMaterial(color: .red, isMetallic: true)
let entity = ModelEntity(mesh: mesh, materials: [material])

entity.position = [0, 1.2, -1]

content.add(entity)

Let's walk through it.

Create the MeshResource to define the shape of the model. You can use MeshResource to create primitive shapes, 3D text and more.

Give the mesh a material to control how it looks. SimpleMaterial is easy to work with, but there are more materials including ones for video and occlusion. You'll find docs for them next to the docs SimpleMaterial.

Create the ModelEntity and add it to the view.

State-of-the-Art 3D (no AR) on macOS using RealityKit?
 
 
Q