Meet Reality Composer Pro

♪ Mellow instrumental hip-hop ♪ ♪ Eric Chiu: Hi there. My name is Eric, and today I'll be giving a walkthrough of Reality Composer Pro. You might remember in 2019, we introduced Reality Composer to get your feet wet with augmented reality. Now, with the release of xrOS, we're introducing Reality Composer Pro, a pro developer tool to help you compose, edit, and preview 3D content. Using Reality Composer Pro, we can create immersive 3D experiences. Let's take this interactive Dioramas app as an example. We can check out fantastic national parks at room scale, along with some points of interest. In this session, we'll go through various features in Reality Composer Pro that we can leverage to build this scene. We'll begin with a fresh project, then go through the general UI of Reality Composer Pro, compose our main scene using a 3D terrain and location pins, add some clouds on top of our terrain, and a few birds with audio. We'll take advantage of statistics to optimize the performance, and finally check out an on-device preview of our scene. Let's get started. First, setting up our project. There are two ways to set up. One way is to go to Xcode > Open Developer Tool and open Reality Composer Pro from there. This is recommended if we want to create a standalone Reality Composer Pro project. The other way is to create a new Xcode project using the xrOS app template. The template automatically generates and links a Reality Composer Pro project to the Xcode project. This is recommended if we want to use a Reality Composer Pro project in an Xcode app. We'll build our Dioramas app this way.

The Reality Composer Pro project is this Swift package. Swift packages not only make it easy to work with Reality Composer Pro, but also during the build process, it packages the content into a framework for our Xcode app to use. Let's take a look at the structure of our Reality Composer Pro project in Finder. Notice here, the files we use to edit 3D content are USDs, otherwise known as Universal Scene Descriptions. This is Apple's preferred file format for 3D content. To learn more, check out the WWDC sessions on the USD file format, listed below. When organizing files, any USD placed in the .rkassets folder will be compiled into a framework, ready to use in our Xcode project. Going back to Xcode, if we select the package, a 3D preview of the default scene appears. If we want to investigate or edit the scene further, all we need to do is click Open in Reality Composer Pro. Now let's go over how to navigate around Reality Composer Pro and get a sense of the UI. In the center, we have the viewport. The viewport allows us to visualize, navigate through, and preview the 3D objects in the scene. We support moving through the viewport using W, A, S, D keys as well as the arrow keys. If we want even more control of the viewport, we can plug in a Mac-compatible game controller and fly through the viewport that way.

On the leading side, we have the hierarchy panel. The hierarchy panel provides an easy way to search, select, and organize the 3D objects in the scene. On the trailing side, we have the inspector panel. The inspector panel provides an easy way to edit the properties of selected objects such as the position, rotation, and scale.

The Add Component button on the bottom of the inspector panel also lets us view the built-in RealityKit Components that we can configure on our objects. Below we have the editor panel. The first tab on the editor panel is the Project Browser. The Project Browser is a user-friendly way to navigate through the files included in the Reality Composer Pro project. This panel also contains other tabs for advanced editing: Shader Graph, Audio Mixer, and Statistics. Now that we have the Reality Composer Pro project set up and know how to navigate around the UI, let's start composing a scene. There's three ways to add assets into a Reality Composer Pro project. The first is importing assets into the Project Browser that already exist on your computer. Let's import the assets we plan to use for our Dioramas app. We can do this by pressing the import button in the Project Browser.

Great! We have all our assets in the Project Browser now. The second is the Content Library, which is a curated library of assets provided by Reality Composer Pro.

The third is Object Capture, which turns photos of real-world objects into 3D models. To learn more, check out the "Meet Object Capture for iOS" WWDC se ssion. Now that we have our assets imported into our project, let's get started on building our scene. First let's delete the sphere created by the template by selecting the object in the viewport and hitting Delete. In the Project Browser, we have a 3D model of Yosemite and a nice diorama base to hold it. We'll drag these two assets into the hierarchy panel, which adds references to those assets in the scene. The diorama base doesn't seem to have a material. Let's check out the Content Library and see if there's any materials we would like to use. Let's click the plus button in the toolbar to open up the Content Library and look for a possible material to use. This concrete material looks promising, so we'll drag it into our scene. The asset is automatically imported into our project and referenced in the scene. Now let's select the diorama base's material in the hierarchy and update it to use the concrete material.

Great! Next, placing some location pins. We'll drag and drop a location pin into the viewport and place it on El Capitan.

Then duplicate the location pin two more times, using Command-D, and rename the three pins to El Capitan, Cathedral Rocks, and Merced River. Let's move the Cathedral Rocks and Merced River location pins to the right spots.

Now that we have the location pins in the right spots, let's group the location pins for organizational purposes and rename the group, "Yosemite_Location_Pins." What if we want to update the location pins to a newer version? Well, all these location pins are referencing the location pin file we previously imported. Leveraging the power of USD, if we ever wanted to update the Location_Pin asset, we can import a new Location_Pin asset into our project and replace the existing one.

Now let's go over particle emitters. Particle emitters allows us to create effects such as this flame in our scene. A particle emitter is composed of two parts: particles and an emitter. Both the particle and emitter have multiple sections of variables to tweak and modify. On the particle, there's the color, properties, force fields, and rendering sections. On the emitter, there's the timing, shape, and spawning sections. Now let's add some clouds using particle emitters to make our scene more interesting and dynamic. How do we do that? Well, our Reality Composer Pro project can contain an arbitrary amount of scenes. Scenes can represent either a finished piece ready to use in our Xcode app, or a reusable piece of content. In this case, clouds are made up of smaller clouds. We'll first build a reusable cloud chunk scene.

Then we'll create a complete cloud scene that combines multiple cloud chunks placed at different positions in 3D space. We'll make three different variations of the complete clouds and finally, put those three clouds into our main scene. Jumping back into Reality Composer Pro, let's create a new scene by going to File > New > Scene. We'll name the scene "Cloud_Chunk". When a new scene is created, it appears as a new tab in the window. There's two ways to add particle emitters to the scene. The first is through the plus button at the bottom of the hierarchy panel. The second is adding a Particle Emitter Component to any object in the scene by clicking the Add Component button at the bottom of the inspector panel. We can preview the particle emitter by using the playback controls near the top of the inspector panel. It's a little hard to see so let's go to Settings and change the viewport background to make the particle emitters more visible.

That's a lot better. Particles are really fun to play with. I often get the results I'm looking for just by experimenting with properties here and there. There's no right or wrong approach to making particle emitters, as long as you get them to a state that pleases you. One thing to consider though is that a high count of particles in a scene may have performance implications, so I would recommend paying close attention to the number of particles being used. Now the particle emitter component has two sections: particles and emitters. In the particles section, we control the appearance of the individual particles. In the emitters section, we control how the particles come out of the origin location. There are also some presets for us to build on top of. so we don't have to start from scratch. The cloud is closest to the Impact preset so let's select the Impact preset to start. My typical workflow is first focusing on the Emitters tab, and then moving on to the Particles tab. Looking at the Emitter tab, Timing is the first section. There's a slight pause in between every emission of particles. This is due to the idle duration variable. Setting this to zero should give a constant emission of particles. Moving on to the Shape section, clouds are more of a sphere shape than a cylinder shape, so let's switch the emitter shape to Sphere. Next, clouds should have particles within its volume, so let's change the birth location from Surface to Volume. Clouds also tend to be more flat than tall, so let's adjust the x- and z-axis scale to be greater than the y-axis scale.

That's looking great. Finally we should check isLocalSpace so that future translating, rotating, and scaling on the particle emitter's parent will also apply to the particle emitter itself. Moving on to the Particles tab, we'll adjust the looks of the individual particles. The birth rate defines the number of particles that exist at once. Being wary of performance, let's crank this down from 2000 to 500 and see if it makes a difference. There's a slight change in appearance, but it still looks pretty good, so let's keep it at 500. Thanks to using the Impact preset, the color and texture of the particles are already what we want, so we can skip modifying variables in those sections and move on to the Properties section. The particles seem to exist for too short of a time. Let's increase the life span from two to five seconds and see what happens.

This looks great. There are more variables we can play with in the Motion, Force Fields, and Rendering sections, but the effect we have so far is close to what we want, so let's not modify those sections for now. We'll save the scene using the shortcut Command-S and move on to the next step. Cool! We just built our cloud chunk that we're going to reuse. Now let's create a new scene that combines multiple cloud chunks to make a complete cloud. We'll create a new scene using the shortcut Command-N. Let's name this scene "Cloud_A." I'll reference the cloud chunk three times by dragging the Cloud_Chunk asset from the Project Browser into the hierarchy panel.

We can select the parent of these three cloud chunks and press play in the playback controls at the top of the inspector panel to preview the cloud. Now let's use the manipulator to move these three cloud chunks to different positions in 3D space.

Amazing. This looks like one cohesive and complete cloud. We can save the scene using Command-S again. In order to get a nice variety of clouds, I made two more clouds using a similar process to making Cloud_A but positioned the cloud chunks differently. I named these scenes Cloud_B and Cloud_C. Now let's jump back into our main scene. I made a group called "Clouds" and imported Cloud_A, Cloud_B, and Cloud_C underneath this group. I also arranged them nicely on top of our terrain. To preview the particle emitters at once, all I have to do is click the group "Clouds" and press play on its playback controls at the top of the inspector panel. Awesome, things are coming together. Next, let's go over how to incorporate audio into our scenes. We can add audio using audio files, audio file groups, and audio sources. Audio files imported into a scene from the Project Browser can be configured in various ways, such as if they should loop. An audio file can be played on one or more objects, and an object can play one or more audio files.

An audio file group can be constructed from audio files in a scene. Each time an object plays an audio file group, a random audio file from the group is selected for playback.

An audio source defines the way audio files and audio file groups are emitted into a scene. There are three kinds of audio sources: spatial, ambient, and channel. Spatial audio sources provide a position and a direction in 3D space for the audio playing. Imagine using spatial audio for a radio asset sitting on a table. Ambient audio sources provide no position but a direction in 3D space for the audio playing. Imagine using ambient audio for wind sounds coming from the east, and no matter how far we walk toward the east, the wind sounds will always be coming from that direction. Finally, channel audio sources provide neither a position nor a direction in 3D space for the audio playing. Imagine using channel audio for background music in an environment. Now that we've covered the audio authoring concepts, let's see it in action in our Dioramas project. We'll create a new scene as a reusable asset that combines a 3D model of a bird with audio files and an audio source. Command-N, and name this scene "Bird_With_Audio." We previously imported a bird with animation and two bird call audio files into our Project Browser. Let's add those assets as references into this scene by dragging them into the hierarchy panel. Now any audio source in the scene can utilize these two bird call audio files. Next let's add a spatial audio source by using the plus button at the bottom of the hierarchy panel. We can also add a spatial audio source to any object in the scene by clicking the Add Component button at the bottom of the inspector panel, and selecting Spatial Audio in the menu. Again, the Spatial Audio source has a position and a direction, so we should place it near the beak of the bird so it sounds more realistic.

We can preview the Spatial Audio source with the playback controls at the top of the inspector panel.

There's additional properties on the spatial audio component we can adjust, such as the focus and which bird call audio file to preview.

Now what if we want to randomly select one of these two bird calls every time the spatial audio source is told to play? Well, that's what audio file groups are for. We can create audio file groups using the same plus button at the bottom of the hierarchy panel. Let's drag in these audio files into the audio file group and name the group "Bird_Calls." Now every time we preview the Spatial Audio source, one of the bird calls within the audio file group is randomly selected to play. Now that we've finished making our reusable asset, let's save and close this Bird_With_Audio scene and open back up our main scene. I arranged five references of Birds_With_Audio in a V-shaped pattern under a group called "Birds." We can select the group "Birds" and press the playback controls in the inspector panel to preview the animation and audio for all the birds at the same time. Now that looks and sounds pretty nice! Using this same procedure, I added the Forest_Sounds audio file into our main scene, along with an ambient audio source named "Nature _Ambience." To play audio in an Xcode app, some additional code is needed, which will be covered in the "Work with Reality Composer Pro content in Xcode" WWDC session. Now that we have the content in our main scene set up, let's go over how to use statistics. Statistics is a dedicated editor for helping us optimize the performance of a scene. The categories are split into general, physics, animation, particle emitters, audio, materials, geometry, and textures. Let's try out statistics in our main scene. If we click on the Statistics tab in the editor panel, the collection of categories appears. The expand button on the top-right corner of every category opens the category up to a more detailed view. Clicking on the expand button for Geometry, we see that there are roughly 464,000 triangles used in the scene, and the Diorama_Base uses the most amount of triangles -- 256,000. We caught something here. The diorama base has a lot more triangles than our 3D terrain, even though its visual shape is less complex than our terrains. This is a great time to simplify the geometry of our diorama base in a 3D modeling software. Luckily, my colleague Scott Wade, who has created the assets for our Dioramas app, has helped me with that. We can import the new simplified version of our diorama's base into our project using the Project Browser's import button. Select our new Diorama_Base, and replace the existing one. Now let's go back to Statistics and take a look at the Geometry category.

The Diorama_Base now only uses 2,500 triangles. We reduced the number of triangles in the scene from 464,000 to 210,000—that's more than half! And our scene maintains the same look prior to our change. As we just saw, Statistics is a great tool to help optimize the performance of a scene. There's one more thing. Let's take a look at how we can preview our scene on device. When we connect our headset to the computer, select it in the dropdown next to the on-device preview button in the toolbar, then toggle the on-device preview button. We should see our scene appear before our very eyes in AR. We can pinch and drag to rotate the scene, and zoom to scale the scene. This makes it very easy to get a sense of what our 3D content looks like in the intended xrOS platform. In interest of time, I performed similar steps of what we went through in this session for another national park, Catalina Island. Using this same workflow, I added an additional 3D terrain of Catalina Island to our main scene. I also added a few location pins to mark Two Harbors, Parsons Landing, and Ribbon Beach on Catalina Island. Finally, I added an Ocean_Sounds audio file for the Nature_Ambience audio source to use. What you saw today is a walkthrough of Reality Composer Pro, a powerful tool to help you assemble 3D scenes. You can add interesting RealityKit Components, such as particle emitters and audio sources. It also helps you optimize your scene and preview it on headset. To learn more about how you can use the Shader Graph to transition between the Yosemite and Catalina Island terrains, watch Neil's session, "Explore materials in Reality Composer Pro." You can also learn how to make this experience interactive in Amanda's session, "Work with Reality Composer Pro content in Xcode". I'm excited to see what you build with Reality Composer Pro. Hope you enjoyed this session and thanks for watching! ♪