Create 3D workflows with USD

Hello and welcome to WWDC. My name is Doug, and today my colleague Nate and I will take a look at how Universal Scene Description, or USD, can be used for a wide variety of 3D work flows.

We'll talk a little bit about what USD is, take a look at the exciting developments n the world of USD, show some incredible new features in Preview, and demonstrate how USD can be at the center of a creative 3D workflow.

USD is a set of libraries and file formats that allows for assembly, interchange, and organization of any number of assets into scenes, shots, or virtual objects. From object capture, asset creation, layout and editing to rendering quick thumbnails for sharing or feedback to creating USDZ assets for use in Augmented Reality, or AR. The flexibility of USD can be a core component of an extensible, collaborative, 3D pipeline.

USD is a foundational technology that, with the growing and deepening integration into Digital Content Creation tools, is enabling more and more ways of creating assets and content. Render, collaborate, for games, AR, film all with USD at the center.

USD was developed by Pixar to enable them to create the enormously complex movies we love. We at Apple worked with Pixar a few years ago to develop the USDZ format to utilize USD in a single file to deliver rich AR content directly to your devices. USD-based workflows are highly collaborative, enabling you to work independently on parts of an asset, easily incorporating your changes with others. It's inherently extensible by design and is rapidly emerging as a key workflow technology. It's built on decades and decades of production experience in the film industry and is increasingly being adopted for games, simulation, AR, manufacturing, and e-commerce. Pixar is using an open and collaborative model that the USD project has been following. The USD forum is active and full of core contributors answering queries, and the Academy Software Foundation has a working group that meets regularly to gather input from interested groups.

Let's take a high level look at other formats out there today.

The most basic format is .obj, which often essentially contains just a single 3D model. It has limited support for materials and no support for animations. Then there's a large group of more modern file formats. These usually support multiple models that can be laid out in a scene graph and have varying levels of support for material and definitions. Many are tied to proprietary tools.

USD supports all of this and is additionally designed to be highly scalable. Pixar developed USD for its use in films, so representing millions of objects is the typical case. And USD is built with collaboration as a core feature, allowing for many artists to work on a single scene without getting in each other's way. USDZ is the archive package for USD and inherits most of these features, is optimized for sharing, and allows a single file to contain all the resources needed for rich AR interchange.

USD naturally allows creation of USDZ files for AR content. But it's also the front-end for a full featured pipeline that delivers USDZ for AR and e-commerce and assets for offline rendering. Last year, we released RealityConverter, a stand-alone utility to allow easy creation of a self-contained USDZ from collections of USD files, .obj, alembic, and a wide variety of other 3D formats as well. On the Apple Developer site, there are command line tools to allow you to automate processing of batches of assets. As the delivery format for AR, USDZs can be authored to target a specific Apple device, letting you tailor the experience for your audience.

The USD ecosystem is growing rapidly. It's really exciting to see the things growing from the roots of what USD provides. Multiverse from J-Cube brings the full power of non-destructive USD editing to the Mac, enabling insanely fast multi-DCC workflows as well as editing in the Multiverse USD Standalone Editor, all powered by the Metal Hydra Storm renderer. OTOY's Octane renderer can plug into DCC's like Houdini and harnesses the blazing-fast power of the GPU to render spectrally correct images directly from USD files. ZBrush works directly with USD mesh data to bring all the power of sculpting to a USD workflow. And Maya continues to integrate USD deeper and deeper, now using USD Hydra rendering directly in the VP2 viewport.

Maya 2022 is the first version to have native seamless integration of USD. This means you can load gigabytes of USD data in seconds and work directly with that data using Maya's native tools. USD data is integrated right into the Attribute Editor and the Outliner. It brings all the USD benefits right inside the DCC: nondestructive editing, variants, and collaboration to name a few. Apple worked with Autodesk to improve the USD importing and exporting functionality as well, including Blendshape support and using textures directly in USDZ files. Of course, USD isn't a static technology. Apple is collaborating with Pixar and the rest of the USD community to extend and improve USD's support for the latest features in 3D content creation. Last year Apple, Nvidia, and Pixar jointly developed a schema to represent properties needed for Physics simulation. It can represent properties from real-time AR physics all the way to offline simulations for robotics and film.

It's exciting to see such widespread support and so many great updates to the USD ecosystem. Another key feature of USD is Hydra.

Hydra is USD's high-performance, modern rendering architecture designed specifically to get millions of objects to your screen as fast as possible. Hydra Storm is one such renderer that is part of the core USD open-source distribution. We're thrilled to share that Preview and Quick Look in macOS will include a Metal-accelerated Hydra Storm renderer. This allows anyone to quickly view USD files in a way that's consistent with Hydra Storm on other platforms all without having to open a DCC. It can handle complex production-grade assets and has some 3D scene exploration tools built in.

Here's a layout asset from Pixar's "Toy Story 4," the antiques mall rendering in real-time in Preview. It has tens of thousands of individual objects and about 34 million triangles. It's a great example of a production-quality asset, and it's a real challenge not only for the renderer but for also any scene navigation tools.

To help move around the scene, any cameras in the USD file are shown in the thumbnail views to the left, letting me click through them. As well as the normal camera controls, I can click on objects in the view and lock the camera to them. To get a more detailed picture of the composition of the scene, we added an outliner that displays all of the object nodes. If I select an object in the view, then it is highlighted in the outliner. But as you can see from this huge list here, we needed to provide better tools to assist with scenes of this complexity. So we've also added the ability to search for objects by name. For example, if I search for all the jukeboxes in the scene, I'll get a list of them and can focus on each in turn.

Once I've found the one I'm interested in, it can be saved as a separate USD file. And then I can view this again in Preview and Quick Look and use it in my AR scene in Reality Composer.

On top of the features I've just shown you, Preview also enables you to render animated USD scenes to a movie file or render the current view to an image, complete with alpha channel.

For AR, Preview uses the RealityKit rendering engine to view USDZ files. It's the same renderer as AR Quick Look on iOS and is a great way to check what an on-device view will look like.

We've worked with Pixar Animation Studios to add Metal and M1 acceleration support to their powerful GPU renderer for producing 3D movies like "Soul" and the upcoming movie "Luca." Pixar Animation Studios is making Metal-accelerated Hydra Storm available for integration in your apps through their USD Open Source project by the end of 2021.

Now let's see how all these exciting USD tools can be used in a single workflow. So let's let Nate get to work.

Thanks, Doug. We're going to show you how you can use USD to capture a real-world object and bring it into 3D. We'll edit it along the way and then render a gorgeous photoreal product shot all with USD.

In a traditional 3D workflow, sharing data between DCCs is cumbersome and error-prone. Lengthy import and export steps are required to move data from one tool to another. This traditional workflow is also one-way. As only the last stages of the pipeline can see the overall scene, earlier stages lack context on how their contributions will be used. In contrast, USD allows us to work in parallel using different tools and to work natively on the same data. Crucially, since all of our data is represented as USD, the import/export step is eliminated and sharing is automatic. A USD workflow enables faster iteration, better collaboration, and opens up new and more creative possibilities. Since it's morning somewhere, we can assume that it's time for breakfast. So let's make some pancakes. Here's a sketch of how we'd like our overall scene to look.

We've storyboarded a few different concepts, camera angles, and layouts, and decided this is our favorite. Let's see how we can use USD to turn this sketch into reality.

Since we're using USD, we can arrange the scene into several separate files. Each file will contribute to some aspect of the scene, be it the layout of the models, sculpting of the geometry, or any animation or lighting we might want to add. These files are all referenced, or linked, into the main scene. This is a core feature of USD which allows any number of people or tools to collaborate on the same scene at the same time. So first, let's set the table. We're using Autodesk's Maya to construct our overall scene. We need a USD file to store all of the layout data. This is where we assemble all of our assets and place them into the scene.

We also want a cool camera move, so we add a USD file to our scene for the camera animation.

To create the 3D model of the pancakes, we'll use Object Capture on our real pancakes. After we've captured the model, we'll use Pixologic's ZBrush to do some cleanup and make some enhancements. Here we've added a sculpting USD file to the scene where this data will live. Can't have pancakes without syrup. We make our virtual syrup in SideFX Houdini, using its fluid simulation capabilities and native support for USD.

Finally, we'll also use Houdini to add some virtual lights to our scene.

Let's see in a little more detail how all of this comes together.

First, we use Object Capture to create a 3D model from our real, physical pancakes. Using a series of photographs taken from different angles, we can construct a 3D mesh and surface material of the pancakes.

Once we've acquired this model, we can view it in Preview. Here's a breakdown of the captured model with and without textures. We'll use this as the basis of our sculpting USD that we'll edit later in ZBrush.

While we're capturing the pancakes, we can use Maya to construct the initial set of USD files where all of our data will live.

As of Maya 2022, USD is natively supported, and it's easy to create a setup like this.

Once we've set up our files and linked in our captured pancakes, we can do the layout. Here I've assembled some of the props that make up the scene. I just need to make a few more adjustments.

This is the breakdown of our scene after we've done the layout. While Object Capture produces excellent results, we weren't able to capture all of the angles of our pancakes before we ate them, so let's take them into ZBrush for some digital enhancement.

Using ZBrush, we can paint in the missing geometry at the bottom of the stack of pancakes where they rested on the plate when we photographed them. In addition, we can add even more physical detail to the model, as well as make some improvements to the surface material. These edits are being made in the sculpting.usd file we created earlier. Since the main scene is referencing this file, everyone on the team will get these updates as they are made.

Object Capture looks great, but you can see the improvement that ZBrush brings by adding detail and polish to the model. While they already look scrumptious, they could use just a little bit more sweetening. For this task, we'll take our pancakes into Houdini and add our virtual syrup and lighting.

In Houdini, we can directly load all of our USD files as native data. Houdini has powerful procedural tools to generate all sorts of effects, and we can use it to pour virtual syrup all over our stack of pancakes.

Here's what our scene looks like with contributions from Maya, ZBrush, and Houdini. Everything is brought together, and now we just need to share it.

Fortunately, we're using USD, and we have a lot of options. We can create a USDZ file and share our scene in Augmented Reality using AR Quick Look on iOS. On the Mac, we can use the new built-in features of Preview that Doug demonstrated earlier. Or we can use a studio-level offline renderer to produce a photo-real video.

First, we'll bring our pancakes into AR. Reality Composer lets us load our USD scene and make any final adjustments before we save it as an AR-enabled USDZ file.

Let's load this up on the iPad Pro and have a look.

Just tap on the USDZ file, and AR Quick Look opens up.

In seconds, we've created pancakes in AR. Looks delicious.

Next, let's check out our scene in Preview on the Mac. We can navigate the scene, inspect objects, select assets, export them, and share. Everything stays as USD.

Finally, we'll create a video using OTOY's Octane to create an offline, path-traced render of our scene. Whatever changes we make to adjust the lighting or materials can be shared back in AR or viewed in Preview. USD makes this all possible. From sketch to reality, thanks to USD and some great tools, we've turned real pancakes into virtual breakfast.

We hope we've given you a glimpse of how you can use USD to create exciting and complex 3D scenes and assets. All of these software vendors are actively integrating USD into their products, so we encourage you to get the latest and greatest and have fun exploring the universe of USD. For more info on Object Capture, check out this session from this year's WWDC. We also have several talks about USD on the Apple platform. Check out this talk on "What's new in USD" from last year's conference. Thank you. [music]