How to store ARKit camera data to calculate projectPoint later?

I have some working sample code, which projects a 3D point back to screen position:

       let pt = camera.projectPoint(worldPosition,
        orientation: .portrait,
        viewportSize: viewportSize)

However, as ARKit camera is moving, the 3D mesh will be updated later, I would like to "store" the ARKit camera data and calculate the final mesh with the earlier cameras. There are 2 approaches I tried:

1. Tried to store the whole camera and restore it later, but I couldn't figure out how to do it
2. Store the camera projectionMatrix and viewMatrix and run my own calculation later. But when I run my projectPoint, the projection is still not 100% correct. This is how I get projectMatrix and viewMatrix:

       let projectionMatrix = camera.projectionMatrix(for: .portrait, viewportSize: camera.imageResolution, zNear: zNear, zFar: zFar)
      let viewMatrix = camera.viewMatrix(for: .portrait)