translate between NSView and MTL drawable coordinates

The viewport is implicitly involved here, but it covers the entire drawable by default, so you don't have to change it unless you want to.

Ultimately, you need to return clip space coordinates from your vertex function. How you get there is up to you, but I find it intuitive to work mostly in view-relative coordinates in app code, then translate to clip coordinates with an orthographic (parallel) projection matrix in the vertex function. Coordinates in Cocoa views assume a lower-left origin unless the view is explicitly flipped. In Metal textures, the origin is at the top-left, so we have to compensate for that by flipping the coordinates ourselves, and also make sure that the scale factors agree.

The details will vary based on your view and layer hierarchy, but supposing you want to translate a mouse event into the coordinates of a CAMetalLayer, you might do this:

// Translate from event's window-relative coordinates to our view
NSPoint pointInView = [view convertPoint:[event locationInWindow] fromView:nil];
// Translate into the pixel-based coordinates of the backing layer
NSPoint pointInBackingLayer = [self convertPointToBacking:pointInView];
// Flip the y-coordinate to move from un-flipped NSView coordinates to Metal's top-left origin convention
NSPoint pointInDrawable = NSMakePoint(pointInBackingLayer.x, metalLayer.drawableSize.height - pointInBackingLayer.y);

This results in a point inside the rect (0, 0) to (drawableSize.width, drawableSize.height). You can pass it into your vertex function in whatever way you find most convenient.

Metal's clip space is a left-handed hemicube which runs from -1 to 1 going from left-to-right and bottom-to-top, and from 0 to 1 from near to far. We can construct a matrix that transforms between an arbitrary 3D volume and this clip space,

matrix_float4x4 matrix_ortho(float left, float right, float bottom, float top, float nearZ, float farZ) {
    return (matrix_float4x4) { {
        { 2 / (right - left), 0, 0, 0 },
        { 0, 2 / (top - bottom), 0, 0 },
        { 0, 0, 1 / (farZ - nearZ), 0 },
        { (left + right) / (left - right), (top + bottom) / (bottom - top), nearZ / (nearZ - farZ), 1}
    } };
}

then build an orthographic matrix each frame that take us from the drawable-relative points to Metal clip space,

matrix_float4x4 projection = matrix_ortho(0, drawableSize.width, drawableSize.height, 0, 0, 1);
[renderEncoder setVertexBytes:&projection length:sizeof(projection) atIndex:1];

and finally transform by this matrix inside the shader:

vertex VertexOut transform_vertex(VertexIn in [[stage_in]],
                                  constant float4x4 &projection [[buffer(1)]])
{
    VertexOut out;
    out.position = projection * in.position;
    return out;
}

This essentially maps each point in the layer to a corresponding point in clip space, which then gets mapped back (via the default viewport) during rasterization to the same point in the layer. If you understand this example thoroughly, you should be able to adjust it to your needs.