How do I use an asset catalog with RealityKit textures?

Context: I have a .nonAR RealityKit scene with a grounding plane. The plane uses a PhysicallyBasedMaterial and loads/applies the textures when the scene starts.

Problem: I'm getting wildly different visual results depending on if I load from a folder in my app bundle or an asset catalog. And, when loading from an asset catalog, my frame rate drops 50% or more.

Question: Should I be using a catalog for RealityKit textures or should I just add them to my project directly? If I should be using an asset catalog, what settings should I apply to the textures for proper rendering?

Rendering samples:

Loading from files I added directly to the project, which gives the most correct looking render ➡️

Loading from an asset catalog with different compression settings ➡️

Up vote post of armthethinker Down vote post of armthethinker
311 views
Posted by
armthethinker
Reply
Add a Comment

Accepted Reply

Hi, we would like to investigate this further, but need some clarification

  1. Which element of the screenshots is the grounding plane?
  2. How exactly the textures are loaded and the PhysicallyBasedMaterial is created?
  3. Why the first screenshot is “most correct” (it looks like there are big alpha’d-out holes)?
  4. What settings are used in Xcode when setting up the asset catalog?

If you are able to give us a sample project that reproduces this issue, that would help us with debugging. You can either upload the project to GitHub/etc and link it here, or file a bug report on feedback assistant with the project attached.

Posted by
Graphics and Games Engineer
Up vote reply of Graphics and Games Engineer Down vote reply of Graphics and Games Engineer
Post marked as solved

  • 1/3. The plane with the gold and black tiling is what I'm focused on. (The content in the middle is a reprojected depth video and can be ignored for my question.) The first image shows what appears to be the right application of the PBR textures to the PBR material to the ground mesh square. The other two images show what happened when I loaded the same textures from the asset catalog. There's also an hdr environment texture for the lighting.

    2. Nothing special. I do var material = PhysicallyBasedMaterial(). I load each texture (albedo, ao, normal, metalness, and roughness) with TextureResource.load(named:), passing in the string of the texture name. Then, for each texture, I apply it to the material with the appropriate parameter (e.g. material.baseColor = PhysicallyBasedMaterial.BaseColor(texture: albedo)). Then, I apply the material to my mesh with ModelEntity(mesh: mesh, materials: [material]). This technique was the same for both loading from an included folder in my app bundle and when I was loading it from the asset catalog.

    4. I tried different combinations of the "Render As" setting and "Compression" setting, but never got the right look. I don't remember the settings exactly as I've since stripped the asset catalog out of my project.

    I'll try to get you all a sample project over the weekend.

    —
    armthethinker

  • Thanks for the information, we are looking forward to a sample project.

    —
    Graphics and Games Engineer

  • Hey! I'm back. I created the repo here: https://github.com/armthethinker/AssetCatalogRealityKit-PossibleBug. There are some notes in the readme that describe the app a bit more. While prepping this, I think the visual error is coming from how the asset catalog is handling the normal map. Hope this helps you all – and I hope that you all can provide a bit of guidance in the future about the best ways to store textures for RealityKit in Xcode.

    —
    armthethinker

Replies

Hi, we would like to investigate this further, but need some clarification

  1. Which element of the screenshots is the grounding plane?
  2. How exactly the textures are loaded and the PhysicallyBasedMaterial is created?
  3. Why the first screenshot is “most correct” (it looks like there are big alpha’d-out holes)?
  4. What settings are used in Xcode when setting up the asset catalog?

If you are able to give us a sample project that reproduces this issue, that would help us with debugging. You can either upload the project to GitHub/etc and link it here, or file a bug report on feedback assistant with the project attached.

Posted by
Graphics and Games Engineer
Up vote reply of Graphics and Games Engineer Down vote reply of Graphics and Games Engineer
Post marked as solved

  • 1/3. The plane with the gold and black tiling is what I'm focused on. (The content in the middle is a reprojected depth video and can be ignored for my question.) The first image shows what appears to be the right application of the PBR textures to the PBR material to the ground mesh square. The other two images show what happened when I loaded the same textures from the asset catalog. There's also an hdr environment texture for the lighting.

    2. Nothing special. I do var material = PhysicallyBasedMaterial(). I load each texture (albedo, ao, normal, metalness, and roughness) with TextureResource.load(named:), passing in the string of the texture name. Then, for each texture, I apply it to the material with the appropriate parameter (e.g. material.baseColor = PhysicallyBasedMaterial.BaseColor(texture: albedo)). Then, I apply the material to my mesh with ModelEntity(mesh: mesh, materials: [material]). This technique was the same for both loading from an included folder in my app bundle and when I was loading it from the asset catalog.

    4. I tried different combinations of the "Render As" setting and "Compression" setting, but never got the right look. I don't remember the settings exactly as I've since stripped the asset catalog out of my project.

    I'll try to get you all a sample project over the weekend.

    —
    armthethinker

  • Thanks for the information, we are looking forward to a sample project.

    —
    Graphics and Games Engineer

  • Hey! I'm back. I created the repo here: https://github.com/armthethinker/AssetCatalogRealityKit-PossibleBug. There are some notes in the readme that describe the app a bit more. While prepping this, I think the visual error is coming from how the asset catalog is handling the normal map. Hope this helps you all – and I hope that you all can provide a bit of guidance in the future about the best ways to store textures for RealityKit in Xcode.

    —
    armthethinker