Metal Draw Single Pixel

I'm trying to use metal to create a pixel art game and I don't know the best way to draw a single pixel. My goal is 1:1 screen to game pixel control so everything looks as sharp as possible.

It seems like drawing primitive points at given vertices results in the game vertices being drawn in between actual device screen pixels despite my best efforts to align them.

I'm wondering if the solution is to draw four vertices around a screen pixel, then using the fragment shader to fill in the color. But that seems like it would take 4 times larger vertex buffers, unless I'm missing something.

Here's an excerpt from my ViewController:

renderEncoder.setVertexBuffer(vertexBuffer, offset:ConstantBufferSize * bufferIndex * sizeof(Float) * 2, atIndex: 0)
renderEncoder.setVertexBuffer(colorBuffer, offset:ConstantBufferSize * bufferIndex * sizeof(Float) * 3, atIndex: 1)
renderEncoder.drawPrimitives(.Point, vertexStart: 0, vertexCount: self.vertexCount, instanceCount: 1)

And the entirety of my shader:

struct VertexInOut
{
    float4 position [[position]];
    float4 color;
    float pointSize [[ point_size ]];
};
constant float POINT_SIZE = 1.0f;
vertex VertexInOut passThroughVertex(uint vid [[ vertex_id ]],
                                     constant packed_float2* position  [[ buffer(0) ]],
                                     constant packed_float3* color    [[ buffer(1) ]])
{
    VertexInOut outVertex;
   
    outVertex.position = float4(position[vid], 0, 1);;
    outVertex.color    = float4(color[vid], 1);
    outVertex.pointSize = POINT_SIZE;
   
    return outVertex;
};
fragment half4 passThroughFragment(VertexInOut inFrag [[stage_in]])
{
    return half4(inFrag.color);
};

Thanks!