Working with USD

Good afternoon. Welcome to our session on Working with USD. It's great to be here. My name is Denis Kovacs and I'm going to be joined on stage by Chloe Moore. Last year, we introduced AR Quick Look on iOS. It's the simplest way to explore 3D content in AR on your devices.

For instance, you can view the model of a product you'd like to buy. You can place it in your own space and get a real sense for its true size. As you move your device around, the model appears firmly grounded in your environment.

And you can also get a good sense of how the real object would look in your home because the model is rendered using real lighting conditions of your space. Shiny materials like the polished chrome on this lamp even reflect your environment.

And AR is simply fun. It's a really intuitive way to explore and interact with your beautiful 3D content.

Underneath all these AR experiences is a file format we introduced last year to our platforms, USDZ.

USDZ support is deeply integrated into our apps and frameworks, so you can share AR experiences with your friends over Messages and Mail, you can experience interactive news with content coming to life in AR, or you can find new experiences on the web, and take them directly into AR through Safari. And you can build your own beautiful apps that take advantage of USDZ.

The technology behind USDZ is USD and it is the main topic of this session.

So here's our agenda for today. First, we will look at what USD is. We'll discuss some workflows related to creating and converting to USDZ. The rest of the talk, we will spend building an intuition for USD. Starting with some essentials common to most 3D file formats and we will end with some features that are unique to USD. So, what is USD? USD stands for Universal Scene Description. It's a 3D file format and a supporting C++ library with Python bindings developed by Pixar. The library can read and write USD files. It contains a powerful composition engine and much, much more. The focus of this library is on speed, scalability and collaboration. There are three file extensions associated with USD, a plain text version, USDA, that is designed to be as easy to read and understand as possible. A binary version, USDC, that is as efficient as possible to read and write. And the extension USD, that can be either plain text or binary. You can go back and forth between these file types.

And then there is USDZ. Let's take a closer look at that. So what is USDZ? USDZ is the distribution format for USD. It contains all the data related to a 3D scene packaged up in a single compact file. It's optimized for sharing and it's the basis of AR Quick Look. USDZ is supported on iOS, macOS and tvOS. If you look at the anatomy of a USDZ file we'll see that it's an uncompressed zip archive where all the contained files are aligned to 64-byte boundaries for most efficient memory mapping. There's two types of files contained in a USDZ archive. The scene description files in USDA, USD, or USDC or even USDZ format so you can have nested archives, and a set of textures currently in PNG or JPEG format.

Let's take a rough comparison of current file formats out there. The basic-- The most basic format is obj, which essentially contains a single 3D model. It has a limited material support and no animations. Then there's a large group of different more modern file formats. They usually can contain multiple models that can be laid out in a scene graph, they can support different sets of material descriptions, and also animation.

USD supports all of this, but additionally it was designed to be scalable, Pixar invented the format for its large movie scenes. And it also supports collaboration, which allows many artists to work together on a single scene without getting in each other's way.

USDZ, as the archive package inherits most of these features as well.

Next, let's take a look at workflows. We can group these in two categories, converting existing assets and creating new ones. Let's start with converting existing assets.

So let's imagine you have your assets on this in some of these other formats, maybe FBX or gltf, and you want to convert them to use USDZ to explore-- deploy them to your app or AR Quick Look.

Today, we introduced a new command line tool called usdzconvert. It supersedes last year's Xcode converter and it allows you to convert many more file formats such as FBX and gltf. It also performs asset validation. So you can be sure that there are no issues with the generated USDZ files. And it is Python based so it's platform independent. So it's easy for you to integrate this converter into your existing content pipelines that might be on other platforms in macOS.

Here are couple of sample sessions. Let's say you have a gltf file, you can pass the file to usdzconvert and it converts it into USDZ. You can also see that it performs asset validation afterwards.

If you want to know all the command line options, you can use this -h. And you can also provide rich material descriptions directly from the command line. This is especially handy for file formats like obj that do not have a rich material definition.

Usdzconvert is a part of a broader package that we are providing to developers. It also includes the precompiled binaries of the USD library for macOS and some other things as well. For example, the USD command line tools. Usdcat, for example, is great to output a plain text representation of your USDZ file. Usdtree is useful to see a high-level structure of the model hierarchy. And usdchecker is USDZ's asset validator. We're also including a little script called fixed capacity. If you offered models with the transparent materials for AR Quick Look last year and those materials look opaque in iOS 13, use a script to fix that. OK. For a little demo of all of these, I'd like to invite Chloe on stage. Thank you, Denis. As we have seen before, we provide a precompiled USD Python library that's available for download here. I have already downloaded and unpacked it. The USD folder contains the precompiled USD Python library, as well as a suite of a command line tools such as usdtree and usdcat provided by Pixar. The usdzconvert folder contains the usdzconvert tool and the fix opacity tool. We also provide a USD command which sets up some environmental variables for you. When you double click it opens up a terminal window.

Let's start with usdzconvert. If you just run usdzconvert by itself, it outputs our argument. Now suppose you have some gltf file that you want to convert.

You can simply run usdzconvert with your file name. Now, let's take a look at a generated asset. To generate to the file contains all materials and animation.

As we can see here, usdzconvert also performs additional asset validation.

Next, let's say you have some obj file without material, like this tetrahedron model.

You can simply add the material with additional arguments, for instance constant color.

Here you can see material is applied.

To see the high-level structure of your model, you can use usdtree.

Here, see the material and the mesh.

To see the full plain text representation of your model, you can use usdcat.

Here's another common use case. You have a obj file here together with a rich set of material textures.

Assigning textures to a material is as simple as assigning constants.

A USDZ file is an uncompressed zip archive. So a quick way to examine its content is to use a zip info. Here we can see the USDZ archive contains a USDZ file and all above textures.

Here you can see all material textures are applied to the model. So to summarize, converting your existing asset to USDZ is easy with usdzconvert. And there is a set of additional tools that would help you to examine your generated assets. Back to you, Denis. Thank you, Chloe.

So that covers the conversion stage. Now let's take a look at how to create USDZ files from scratch.

You can of course start creating your assets with your favorite authoring tool. And it will likely be able to export to a format like FBX already that you can then convert to USDZ with the usdzconvert tool that we just saw.

But there's a growing number of content creation applications that support direct USD export. That allows you to integrate your content creation step more easily into a USD pipeline. And the conversion step with all these potential translation complications turns into a much simpler archiving step.

Here are two examples of content creation tools that support USD export. This one is Substance Painter by Adobe, a great tool for creating photorealistic materials for your 3D objects. It supports USD and USDZ export. And here is Autodesk Maya, one of the leading 3D modeling and animation tools. It supports USD and USDZ export through a plugin that's written and maintained by Pixar. So as we've seen, a lot of these applications even support direct USDZ export further simplifying your USDZ creation step. And because of USD's Python bindings you can also create your own custom Python pipelines that are tailored to your particular content. Next, we're making exporting USDZ from SceneKit super easy. All you have to do is to create or load your SCNScene just as before. And when you're ready to export to USDZ, all you need is one API call, write to, and make sure that the file extension is usdz.

In fact, we've integrated USDZ export into Xcode's SceneKit editor as well. So, you can also export directly through a command-- through a user interface as well. OK. So that covers the workflows.

Now, we want to take a look at USD's features. Now, as we've mentioned in the introduction, USD has a very large feature set. And it is impossible to even mention all of them in this talk.

But there are a few key concepts that are easy to grasp. So let's focus on those first.

We will talk about four concepts, the File Structure, the Scene Graph, Mesh Data, and Materials. A key differentiator of USD is that its plain text form is designed to be readable and understandable. So the format becomes less mysterious and more transparent. And when things go wrong, you can more quickly pinpoint where and why.

So let's focus on the file structure first.

So this is an excerpt of a USDA file. The first thing you will notice is that it contains these nested containers. In USD they're called prims.

These prims contain properties that store the actual data. And there's a set of metadata attached to file level, prim level, or property level. Because of the nested structure, every object can be accessed through an object path. In this case, the prim cube is nested under simple mesh, so its path becomes /simpleMesh/cube.

This also works with properties, with the dot notation.

Now that we have seen the basic file structure, let's take a look at how it is used to store scene data. A scene graph defines the object hierarchy. Transforms on a parent also affect its children. In this case, we have a cube and a sphere parented to the cube. If you move the cube, the sphere comes along. If you move the sphere, the cube stays put. A scene graph is easy to store with a nested prim structure of USD. You can see the ultra prim of the parents with the two child prims, cube and sphere. You can also see that there are other prims such as materials or maybe animations that don't belong to the scene graph.

So that covers scene graphs. Now let's take a look at mesh data. Mesh data can be grouped roughly in two categories. Mesh attributes such as positions, normals, and texture coordinates.

And mesh connectivity, such as the numbers of vertices per face or which vertex indices belong to a face. Let's look at that at the simple example of a tetrahedron. We start with a set of points, four points for the tetrahedron. And each face contains three vertices, so there are four triangles. And for each face, we will also store the vertex indices. Next, we'll store normals. And in this case we choose to store the normals per face. So there are four normals. And the USD language for per-face attributes is uniform. So the metadata is set to uniform.

That's also store texture coordinates. These are going to be sort as indexed, mesh attributes, and the indices are per-face, per-vertex, and in USDs language studies face bearing. Lastly, we'll make sure that this tetrahedron is rendered as a polygon and not as a subdivision surface. We'll come back to that later. So now we've seen the scene graph, and the mesh data, let's take a look at how materials are defined.

USD's material definition is very rich. It is meant for movie quality output. For realistic real-time rendering, there's a smaller subset called UsdPreviewSurface. It's a physically-based material description and it supports two workflows, metallic-roughness and specular-roughness. We will focus on metallic-roughness here. So in this example, the same gramophone that we saw in Chloe's demo, we'll first assign a constant, white color. Next, we will set normal and occlusion maps to bring out the fine details.

Next, let's crank up the metallic level so it becomes really shiny. And we will also add a roughness map to dull out parts of the object. Lastly, we will also set diffuse color and metallic textures. So with these five textures, we get the final beautiful photorealistic look of this mesh. So let's see how this is described in USD. USD uses a shader node graph that has separate shader nodes that are connected to each other. UsdPreviewSurface has a very simple shader node graph structure, with only four node types. The main shader node that defines all the PBR attributes, we've seen five of them before. A texture sampler that tells you which texture to use, a mesh attribute reader for example for the texture coordinates, and a UV transformed node that allows you to scale or rotate your texture coordinates. Let's take a look at the simplest material example with a couple of constant-- constants for the mesh attributes.

We can see here that the main shader node's surface output is wired to the material's surface output. Here is what it looks like in USD. You have the outer nested material prim with main shader node prim inside, and the output of the shader node is wired to the output of the material using these object paths that we've seen before. Next, let's take a look at a slightly more complicated example of a texture assigned to the diffuse color channel. We will need a mesh attribute reader for the texture coordinates, that's the red shader node, a texture sampler for the texture that we want to apply, it's the orange one, and the main PBR shader node with the diffuse color attributes.

This is what it looks like in USD. You have the outer material prim again and then the three shader nodes. And the outputs are wired to some of the inputs of the next shader node. OK. So that covers essentials in USD. All of these concepts that we talked about are in a samples folder where we provided little sample scripts to create scene graphs, mesh data, material and mesh groups, so that you can assign multiple materials to single mesh, and even animations for transform animations and for skinning and skeletal animations.

Now that we covered the basic structure of USD, let's take a look at the main advantages that set it apart from other formats.

Scalability to complex scenes and life composition and collaboration.

For scalability, we will look at something called subdivision surfaces.

And for collaboration, we will look at the composition engine. So let's start with subdivision surfaces.

Subdivision surfaces are an inefficient representation of curved surfaces. In contrast, polygonal surface descriptions are an approximation of the real curved surface. And that approximation bakes in some assumptions on what a good approximation is, for example, how closely you're going to be to the actual object. In contrast, subdivision surfaces described the true surface. So you can do dynamic tessellation, for example based on the distance to the camera to find a good enough approximation of the true surface with small polygons.

Subdivision surfaces are also great for animated content. And we have been using subdivision surfaces in the past couple years, for example, for an emoji or memoji. On the left-hand side, you can see what the subdivision control mesh would look like if you rendered it as a polygonal surface. You can see that it's fairly course, which is great in terms of the memory footprint. On the right-hand side, you can see the final subdivided surface, and you can see the beautiful details in there.

The industry standard for subdivision surfaces is OpenSubdiv and it is also developed by Pixar. We have been working closely with Pixar to provide metal shaders that work optimally on our platforms. So the GPU evaluation is as fast as possible. And OpenSubdiv is the basis for subdivision surfaces in Scenekit. And of course USD has great support for subdivision surfaces and it works beautifully with OpenSubdiv. In fact, USD has a unified description for both polygonal mesh data and subdivision surface data. It also has subdivision specific-- support for subdivision-specific properties, such as creases or corners.

So, that concludes subdivision surfaces, a great and efficient way to store high quality surfaces. Now let's take a look at the composition engine.

The composition engine is a powerful authoring tool that enables efficient collaboration between artists. And we'll take a look at an example of references. Here's again the scene graph with two child objects parented under a parent node. And this is an efficient representation, as long as those two objects are different. But what if you have multiple copies of the same object in your scene graph? Then this nested scene graph structure becomes inefficient because it stores duplicate data.

Most filed formats factor out the duplicate data and store references instead, and use the support set as well. But it goes further. It also creates virtual objects inside this nested scene graph prim structure, and each of these virtual objects has its own unique object path, so that you can go in and override some of its materials-- some of its properties. For example, in this case, I overrode the second spheres material color to yellow without affecting the first spheres material.

You can also remove whole subtrees from your scene graph. And USD stores all of these edits as efficient overrides to the original data. Let's take a look at what artist workflows this enables. This is the finder in macOS Catalina. And you can see here macOS Quick Look rendering a scene, USD file. Let's say now, I'm a layout artist that places all these objects in the scene. And at the same time, there's another artist working on refining some of these objects, for example, this chair. And this artist replaced this chair object with this one. Now, if you go back to the layout file, you can see that these objects have been automatically replaced, the file itself did not change, it pulled in the references. And in fact, I can do my own edits. For example, changing the color without affecting the other artists files. So this allows us to decouple our workflows from each other.

So this was a quick sneak peek on all the new artist workflows that USD can enable.

We have also included two more examples of these more advanced USD features in the samples folder. Subdivisions surfaces with creases and references with overrides.

So, in summary, we have looked at universal scene description. It is a file format and a powerful library for authoring all the way to deploying 3D content.

We have looked at workflows to create and convert assets to USDZ. We introduced a new tool called usdxconvert that converts obj, gltf, fbx and other file formats to USDZ. And you can download these Python USDX tools, which include usdzconver, a precompiled USD library and the sample scripts from our website.

For more information, please check out the session page. And we will have a lab on USD and USDZ tomorrow at 3 p.m. And also check out the advances in AR Quick Look on Friday at 9:00 a.m. Thank you. [ Applause ]