Updating mesh vertex positions per frame (from CPU)

We have a reasonably complex mesh and need to update the vertex positions for every frame using custom code running on the CPU.

It seems like SceneKit is not really set up to make this easy, as the SCNGeometry is immutable.

What is the easiest (yet performant) way to achieve this?


So far I can see two possible approaches:

1) Create a new SCNGeometry for every frame. I suspect that this will be prohibitively expensive, but maybe not?

2) It seems that SCNProgram and its handleBinding... method would allow updating the vertex positions. But does using SCNProgram mean that we have to write all our own shaders from scratch? Or can we still use the default Scenekit vertex and fragment shaders even when using SCNProgram?

Answered by ppix in 284453022

Just in case somebody else with the same problem finds this post:

The solution I found was to use a SCNGeometrySource wrapping a custom MTLBuffer.

We then use another MTLBuffer (actually multiple in a ring-buffer scheme) that we write to from our CPU code. Once the CPU has finished writing the vertex positions, the next time renderer(_:willRenderScene:atTime:) comes around, we queue up a blit shader that copies from this MTLBuffer into the MTLBuffer used by the SCNGeometrySource.

I also wrote a two stage compute shader for computing the vertex normals based on the updated vertex positions. These vertex normals are also written to a MTLBuffer that is wrapped in a SCNGeometrySource.

This scheme works beautifully and seems to have very low overhead in terms of both CPU and GPU load.

Thank you.

Yes, I think option 1 is definitely worth trying. Performance might just be fine.

If not, it gets tricky. Shadermodifier won't help (as far as I can tell), because they do not provide a way of getting the new vertex from the CPU to the GPU.


I don't think updating the vertex colors and then copying the info from color to position on the GPU would buy me anything. Because updating the color information isn't any easier than updating the vertex positions.


When updating the colors by rebuilding the SCNGeometry using that last line of code, how has performance been for you? Has it been an issue? For how complex a mesh? Because that approach is basically the option 1 solution I was thinking about (just replacing vertex positions instead of colors).

Awesome, thank you for trying that out. That is very encouraging!

Sounds like replacing the SCNGeometry every frame (option 1) might be fine then. We are targeting the new iPhones, so our framerate target is just 60fps, and the CPU & GPU performance should be pretty similar. And I believe the mesh will be around 20k triangles or so.

Thanks, yes, I am quite aware of the SCNMorpher. However as I wrote in the question, we really did need to udated the vertex positions using custom code on the CPU. The SCNMorpher only allows you to interpolate between a set of target poses.

Accepted Answer

Just in case somebody else with the same problem finds this post:

The solution I found was to use a SCNGeometrySource wrapping a custom MTLBuffer.

We then use another MTLBuffer (actually multiple in a ring-buffer scheme) that we write to from our CPU code. Once the CPU has finished writing the vertex positions, the next time renderer(_:willRenderScene:atTime:) comes around, we queue up a blit shader that copies from this MTLBuffer into the MTLBuffer used by the SCNGeometrySource.

I also wrote a two stage compute shader for computing the vertex normals based on the updated vertex positions. These vertex normals are also written to a MTLBuffer that is wrapped in a SCNGeometrySource.

This scheme works beautifully and seems to have very low overhead in terms of both CPU and GPU load.

Did you open source this solution? Would love to take a look! Dealing with the exact same issue currently.

Hi, I'm having the same problem here. I'd appreciate it if anyone has an example.
I sorta figured out a way to do this. The key is to use SCNGeometrySource's proper initializer that takes an MTLBuffer as input: https://developer.apple.com/documentation/scenekit/scngeometrysource/1522873-init.

I'm posting my pseudo code here in case anyone comes across the same issue.

Code Block swift/* initialize color buffer, similarly for vertex buffer */var color_buffer_array : [UInt8] = []/* appending rgba values (0...1) to the color buffer */color_buffer_array.append(contentsOf: withUnsafeBytes(of: Float(UInt8(xyzrgb[6])!) / 255.0, Array.init))color_buffer_array.append(contentsOf: withUnsafeBytes(of: Float(UInt8(xyzrgb[7])!) / 255.0, Array.init))color_buffer_array.append(contentsOf: withUnsafeBytes(of: Float(UInt8(xyzrgb[8])!) / 255.0, Array.init))color_buffer_array.append(contentsOf: withUnsafeBytes(of: Float(1.0), Array.init))curr_point_cloud.color_buffer = Data(color_buffer_array)/* NOTE: use 4 UInt8's for rgba in Data, use 4 Floats for rgba in MTLBufferbelow is an example of not using MTLBuffer, so color cannot be updated in real time//    curr_point_cloud.color_source = SCNGeometrySource(data: curr_point_cloud.color_buffer!,//                             semantic: .color,//                             vectorCount: curr_point_cloud.points.count, // number of vertices//                             usesFloatComponents: true, // this has to be true in order to display correct color//                             componentsPerVector: 4, // 4 UInt8's: r, g, b, a//                             bytesPerComponent: 1, // 1 UInt8 == 1 byte//                             dataOffset: 0,//                             dataStride: 4) // 4 * 1*//* below is an example of using MTLBuffer, so color can be updated in real time */curr_point_cloud.color_buffer!.withUnsafeBytes { rawBufferPointer in      let rawPtr = rawBufferPointer.baseAddress!      curr_point_cloud.color_mtl_buffer = mtl_device!.makeBuffer(bytes: rawPtr, length: curr_point_cloud.color_buffer!.count, options: [])      curr_point_cloud.tmp_color_mtl_buffer = mtl_device!.makeBuffer(bytes: rawPtr, length: curr_point_cloud.color_buffer!.count, options: [])      curr_point_cloud.color_source = SCNGeometrySource(buffer: curr_point_cloud.color_mtl_buffer!, vertexFormat: .float4, semantic: .color, vertexCount: curr_point_cloud.points.count, dataOffset: 0, dataStride: 16)}/* update MTLBuffer    // NOTE: two options, change color_mtl_buffer directly, or change tmp_color_mtl_buffer and use blit command    func renderer(_ renderer: SCNSceneRenderer, willRenderScene scene: SCNScene, atTime time: TimeInterval) {      if DataModel.shared.point_cloud_objects.count == 0 { return }      // https://stackoverflow.com/questions/40476426/scenekit-metal-depth-buffer      let commandBuffer = DataModel.shared.mtl_command_queue!.makeCommandBuffer()!      let blitCommandEncoder: MTLBlitCommandEncoder = commandBuffer.makeBlitCommandEncoder()!      blitCommandEncoder.copy(from: DataModel.shared.point_cloud_objects[0].tmp_color_mtl_buffer!, sourceOffset: 0, to: DataModel.shared.point_cloud_objects[0].color_mtl_buffer!, destinationOffset: 0, size: DataModel.shared.point_cloud_objects[0].color_buffer!.count)      blitCommandEncoder.endEncoding()      commandBuffer.commit()    }*/

Updating mesh vertex positions per frame (from CPU)
 
 
Q