Hi, I've got a Swift Framework with a bunch of Metal files. Currently users have to manually include a Metal Lib in their bundle provided separately, to use the Swift Package.
First question; Is there a way to make a Metal Lib target in a Swift Package, and just include the .metal files? (without a binary asset)
Second question; If not, Swift 5.3 has resource support, how would you recommend to bundle a Metal Lib in a Swift Package?
General
RSS for tagDelve into the world of graphics and game development. Discuss creating stunning visuals, optimizing game mechanics, and share resources for game developers.
Post
Replies
Boosts
Views
Activity
I am trying to build a website where I would like to render the USDZ 3D model on the browser without the AR feature. The user should be able to interact with the 3D model using a pointing device (mouse). If the user wants to see the 3D model in AR she/he can do so by loading the page on a compatible device where the 3D model can be projected in AR.
I am looking for an answer to how to display the USDZ 3D model on the browser without the AR feature.
I searched the Metal Shading Language Specification Version 3.0 document, however I cannot see any function for inverting a matrix. Is there really no function in Metal for inverting a matrix?
I often need to this in linear equations and have so far resorted to writing the necessary function each time, most of the time just copy-and-pasting code.
inverse exists in SIMD and GLSL, so why not in Metal? It seems so unexpected that this function does not exist that I am almost certain I have just overlooked something obvious. I even tried 1 / M, to no avail.
Hi,
The metal-cpp distribution appears to only contain headers for Foundation and Quartzcore. The LearnMetalCPP download [1] provides a ZIP with an metal-cpp-extensions directory containing AppKit.hpp and MetalKit.hpp headers. First question: Are these headers distributed anywhere else more publicly? Without these headers only the renderer can be fully written in C++ as far as I can tell, i.e. no complete C++ NSApplication. Second question: Will these headers, if needed, be maintained (e.g. updated and/or extended) by Apple along side metal-cpp?
[1] https://developer.apple.com/metal/cpp/
Thank you and regards.
Guys,
In my main application bundle, I have included a helper bundle in its Resources. When the helper requests Accessibility permission, the system modal window displays what the helper is requesting permission for.
However, when the helper requests permission for Screen Recording, the system modal window displays that the main application bundle is requesting permission, which includes the helper.
This issue seems to be specific to Ventura, as both requests are displayed on behalf of the helper in Monterey.
I'm wondering if this is a known issue or limitation or if there is a way to make the permission request specifically from the helper.
I am running the RoomPlan Demo app and keep getting the above error and when I try to find someplace to get the archive in the Metal Libraries my searches come up blank. There are no files that show up in a search that contain such identifiers. A number of messages are displayed about "deprecated" interfaces also. Is it normal to send out demo apps that are hobbled in this way?
, it is after update to Xcode 14.3:
[default] CGSWindowShmemCreateWithPort failed on port 0
Hi! I am currently trying to upload my iOS app to App Store Connect. Unfortunately, code signing fails with the following error: "Code object is not signed at all.", referencing a binary Metallib (created with metal-tt and an mtlp-json script). I am using Xcode's automatically managed signing and the binary metallib is located inside the "Resources" directory of a framework that I am including with "Embed and sign" in the app. Could anyone give some guidance on what I need to change to make code signing work? Thank you.
I'm really excited about the Object Capture APIs being moved to iOS, and the complex UI shown in the WWDC session.
I have a few unanswered questions:
Where is the sample code available from?
Are the new Object Capture APIs on iOS limited to certain devices?
Can we capture images from the front facing cameras?
Is MTKView intentionally unavailable on visionOS or is this an issue with the current beta?
Sample project from: https://developer.apple.com/documentation/RealityKit/guided-capture-sample was fine with beta 3.
In beta 4, getting these errors:
Generic struct 'ObservedObject' requires that 'ObjectCaptureSession' conform to 'ObservableObject'
Does anyone have a fix?
Thanks
What is the most efficient way to use a MTLTexture (created procedurally at run-time) as a RealityKit TextureResource? I update the MTLTexture per-frame using regular Metal rendering, so it’s not something I can do offline. Is there a way to wrap it without doing a copy?
A specific example would be great.
Thank you!
Hi, I trying to use Metal cpp, but I have compile error:
ISO C++ requires the name after '::' to be found in the same scope as the name before '::'
metal-cpp/Foundation/NSSharedPtr.hpp(162):
template <class _Class>
_NS_INLINE NS::SharedPtr<_Class>::~SharedPtr()
{
if (m_pObject)
{
m_pObject->release();
}
}
Use of old-style cast
metal-cpp/Foundation/NSObject.hpp(149):
template <class _Dst>
_NS_INLINE _Dst NS::Object::bridgingCast(const void* pObj)
{
#ifdef __OBJC__
return (__bridge _Dst)pObj;
#else
return (_Dst)pObj;
#endif // __OBJC__
}
XCode Project was generated using CMake:
target_compile_features(${MODULE_NAME} PRIVATE cxx_std_20)
target_compile_options(${MODULE_NAME}
PRIVATE
"-Wgnu-anonymous-struct"
"-Wold-style-cast"
"-Wdtor-name"
"-Wpedantic"
"-Wno-gnu"
)
May be need to set some CMake flags for C++ compiler ?
I'm using DrawableQueue to create textures that I apply to my ShaderGraphMaterial texture. My metal render is using a range of alpha values as a test.
My objects displayed with the DrawableQueue texture are working as expected, but the alpha component is not working.
Is this an issue with my DrawableQueue descriptor? My ShaderGraphMaterial? A missing setting on my scene objects? or some limitation in visionOS?
DrawableQueue descriptor
let descriptor = await TextureResource.DrawableQueue.Descriptor(
pixelFormat: .rgba8Unorm,
width: textureResource!.width,
height: textureResource!.height,
usage: [.renderTarget, .shaderRead, .shaderWrite], // Usage should match the requirements for how the texture will be used
//usage: [.renderTarget], // Usage should match the requirements for how the texture will be used
mipmapsMode: .none // Assuming no mipmaps are needed for the text texture
)
let queue = try await TextureResource.DrawableQueue(descriptor)
queue.allowsNextDrawableTimeout = true
await textureResource!.replace(withDrawables: queue)
Draw frame:
guard
let drawable = try? drawableQueue!.nextDrawable(),
let commandBuffer = commandQueue?.makeCommandBuffer()//,
//let renderPipelineState = renderPipelineState
else {
return
}
let renderPassDescriptor = MTLRenderPassDescriptor()
renderPassDescriptor.colorAttachments[0].texture = drawable.texture
renderPassDescriptor.colorAttachments[0].loadAction = .clear
renderPassDescriptor.colorAttachments[0].storeAction = .store
renderPassDescriptor.colorAttachments[0].clearColor = clearColor
/*renderPassDescriptor.colorAttachments[0].clearColor = MTLClearColor(
red: clearColor.red,
green: clearColor.green,
blue: clearColor.blue,
alpha: 0.5 )*/
renderPassDescriptor.renderTargetHeight = drawable.texture.height
renderPassDescriptor.renderTargetWidth = drawable.texture.width
guard let renderEncoder = commandBuffer.makeRenderCommandEncoder(descriptor: renderPassDescriptor) else {
return
}
renderEncoder.pushDebugGroup("DrawNextFrameWithColor")
//renderEncoder.setRenderPipelineState(renderPipelineState)
// No need to create a render command encoder with shaders, as we are only clearing the drawable.
// Since we are just clearing the drawable to a solid color, no need to draw primitives
renderEncoder.endEncoding()
commandBuffer.commit()
commandBuffer.waitUntilCompleted()
drawable.present()
}
Hi there -
Where would a dev go these days to get an initial understanding of SceneKit?
The WWDC videos linked in various places seem to be gone?!
For example, the SceneKit page at developer.apple.com lists features a session videos link that comes up without any result, https://developer.apple.com/scenekit/
Any advice..?
Cheers,
Jan
I've been attempting to use the new CAMetalDisplayLink to simplify the code needed to sync my rendering with the display across Apple platforms. One thing I noticed since moving to using CAMetalDisplayLink is that the Metal Performance HUD which I had previously been using to analyze the total memory used by my app (among other things) is suddenly no longer appearing when using CAMetalDisplayLink.
This issue can be reproduced with the Frame Pacing sample from WWDC23
Anyone from Apple know if this is expected behavior or have an idea on how to get this to work properly?
I've filed FB13495684 for official review.
Hi there, I have some existing metal rendering / shader views that I would like to use to present stereoscopic content on the Vision Pro. Is there a metal shader function / variable that lets me know which eye we're currently rendering to inside my shader? Something like Unity's unity_StereoEyeIndex? I know RealityKit has GeometrySwitchCameraIndex, so I want something similar (but outside of a RealityKit context).
Many thanks,
Rich
Hi, I'm displaying linear gray by CAMetalLayer with the shader below.
fragment float4 fragmentShader(VertexOut in [[stage_in]],
texture2d<float, access::sample> BGRATexture [[ texture(0) ]])
{
float color = in.texCoordinates.x;
return float4(float3(color), 1.0);
}
And my CAMetalLayer has been set to linearSRGB.
metalLayer.colorspace = CGColorSpace(name: CGColorSpace.linearSRGB)
metalLayer.pixelFormat = .bgra8Unorm
Why the display seem add gamma? Apparently the middle gray is 187 but not 128.
Hello,
I've been tinkering with PortalComponent on visionOS a bit but noticed that the content of the WorldComponent is always clipped to the mesh geometry of whatever entities have the PortalComponent applied. Now I'm wondering if there is any way or trick to allow contents of the portal to peek out – similar to the Encounter Dinosaurs experience on Vision Pro (I assume it also uses PortalComponent?).
I saw that PortalComponent has a clippingPlane property (https://developer.apple.com/documentation/realitykit/portalcomponent/clippingplane-swift.property). But so far I haven't been able to achieve a perceptible visual difference with it.
If possible I would like to avoid hacky tricks using duplicate meshes or similar to achieve this.
Thanks for any hints!
In the below code I have extracted face mesh vertices from ARKit face anchors and created a custom face mesh using SceneKit SCNGeometry. This enabled me to stretch face mesh vertices as per my requirement.
Now the problem I am facing is as follows. I am trying to apply a lipstick texture material which is of type SCNMaterial. Although ARSCNFaceGeometry lets me apply different textures through SCNMaterial and SCNNode, I am not able to do the same using mu CustomFaceGeometry. When I am applying a lipstick texture which looks like the image attached below, the full face is getting colored or modified, I want only that part of the face which has texture transparency as >0 and I dont want other part of the face to be modified.
Can you give me a detailed solution using code?
// ViewController.swift
import UIKit
import ARKit
import SceneKit
import simd
class ViewController: UIViewController, ARSCNViewDelegate, ARSessionDelegate{
@IBOutlet weak var sceneView: ARSCNView!
let vertexIndicesOfInterest = [250]
var customFaceGeometry: CustomFaceGeometry!
var scnFaceGeometry: SCNGeometry!
private var faceUvGenerator: FaceTextureGenerator!
var faceGeometry: ARSCNFaceGeometry!
override func viewDidLoad() {
super.viewDidLoad()
sceneView.delegate = self
override func viewWillAppear(_ animated: Bool) {
super.viewWillAppear(animated)
let configuration = ARFaceTrackingConfiguration()
sceneView.session.run(configuration)
}
}
extension ViewController {
func renderer(_ renderer: SCNSceneRenderer, didUpdate node: SCNNode, for anchor: ARAnchor) {
guard let faceAnchor = anchor as? ARFaceAnchor else { return }
customFaceGeometry = CustomFaceGeometry(fromFaceAnchor: faceAnchor)
let customGeometryNode = SCNNode(geometry: customFaceGeometry.geometry)
customFaceGeometry.geometry.firstMaterial?.fillMode = .lines
customFaceGeometry.geometry.firstMaterial?.transparency = 0.0
customFaceGeometry.geometry.firstMaterial?.isDoubleSided = true
node.addChildNode(customGeometryNode)
}
func renderer(_ renderer: SCNSceneRenderer, willUpdate node: SCNNode, for anchor: ARAnchor) {
guard let faceAnchor = anchor as? ARFaceAnchor,
let faceMeshNode = node.childNodes.first else { return }
DispatchQueue.main.async {
self.customFaceGeometry.update(withFaceAnchor: faceAnchor, node: faceMeshNode)
}
}
}
class CustomFaceGeometry {
var geometry: SCNGeometry
let lipImage = UIImage(named: "Face.scnassets/lip_arks_y7.png")
init(fromFaceAnchor faceAnchor: ARFaceAnchor) {
self.geometry = CustomFaceGeometry.createCustomSCNGeometry(from: faceAnchor)!
}
static func createCustomFaceGeometry(fromVertices vertices_o: [SCNVector3]) -> SCNGeometry {
var vertices = vertices_o
let vertexData = Data(bytes: vertices, count: vertices.count * MemoryLayout<SCNVector3>.size)
let vertexSource = SCNGeometrySource(data: vertexData,
semantic: .vertex,
vectorCount: vertices.count,
usesFloatComponents: true,
componentsPerVector: 3,
bytesPerComponent: MemoryLayout<Float>.size,
dataOffset: 0,
dataStride: MemoryLayout<SCNVector3>.stride)
let indices: [Int32] = Array(0..<Int32(vertices.count))
let indexData = Data(bytes: indices, count: indices.count * MemoryLayout<Int32>.size)
let element = SCNGeometryElement(data: indexData, primitiveType: .point, primitiveCount: vertices.count, bytesPerIndex: MemoryLayout<Int32>.size)
return SCNGeometry(sources: [vertexSource], elements: [element])
}
static func createGeometry(fromFaceAnchor faceAnchor: ARFaceAnchor) -> SCNGeometry
let vertices = faceAnchor.geometry.vertices.map { SCNVector3($0.x, $0.y, $0.z) }
return CustomFaceGeometry.createCustomFaceGeometry(fromVertices: vertices)
}
func update(withFaceAnchor faceAnchor: ARFaceAnchor, node: SCNNode) {
if let newGeometry = CustomFaceGeometry.createCustomSCNGeometry(from: faceAnchor) {
node.geometry = newGeometry
let lipstickNode = SCNNode(geometry: newGeometry)
let lipstickTextureMaterial = SCNMaterial()
lipstickTextureMaterial.diffuse.contents = lipImage
lipstickTextureMaterial.transparency = 1.0
lipstickNode.geometry?.firstMaterial = lipstickTextureMaterial
node.geometry?.firstMaterial?.fillMode = .lines
node.geometry?.firstMaterial?.transparency = 0.5
}
}
static func createCustomSCNGeometry(from faceAnchor: ARFaceAnchor) -> SCNGeometry? {
let faceGeometry = faceAnchor.geometry
var vertices: [SCNVector3] = faceGeometry.vertices.map { SCNVector3($0.x, $0.y, $0.z) }
print(vertices[250])
let ll_ratio_y = Float(0.969999)
vertices[290] = SCNVector3(x: vertices[290].x, y: vertices[290].y*ll_ratio_y, z: vertices[290].z)
vertices[274] = SCNVector3(x: vertices[274].x, y: vertices[274].y*ll_ratio_y, z: vertices[274].z)
vertices[265] = SCNVector3(x: vertices[265].x, y: vertices[265].y*ll_ratio_y, z: vertices[265].z)
vertices[700] = SCNVector3(x: vertices[700].x, y: vertices[700].y*ll_ratio_y, z: vertices[700].z)
vertices[730] = SCNVector3(x: vertices[730].x, y: vertices[730].y*ll_ratio_y, z: vertices[730].z)
vertices[25] = SCNVector3(x: vertices[25].x, y: vertices[25].y*ll_ratio_y, z: vertices[25].z)
vertices[709] = SCNVector3(x: vertices[709].x, y: vertices[709].y*ll_ratio_y, z: vertices[709].z)
vertices[725] = SCNVector3(x: vertices[725].x, y: vertices[725].y*ll_ratio_y, z: vertices[725].z)
vertices[710] = SCNVector3(x: vertices[710].x, y: vertices[710].y*ll_ratio_y, z: vertices[710].z)
let vertexData = Data(bytes: vertices, count: vertices.count * MemoryLayout<SCNVector3>.size)
let vertexSource = SCNGeometrySource(data: vertexData, semantic: .vertex, vectorCount: vertices.count, usesFloatComponents: true, componentsPerVector: 3, bytesPerComponent: MemoryLayout<Float>.size, dataOffset: 0, dataStride: MemoryLayout<SCNVector3>.stride)
let indices: [UInt16] = faceGeometry.triangleIndices.map(UInt16.init)
let indexData = Data(bytes: indices, count: indices.count * MemoryLayout<UInt16>.size)
let element = SCNGeometryElement(data: indexData, primitiveType: .triangles, primitiveCount: indices.count / 3, bytesPerIndex: MemoryLayout<UInt16>.size)
return SCNGeometry(sources: [vertexSource], elements: [element])
}
}