MetalKit masking two quads

Hi atirek123, could you explain a little more what you are trying to do? I'm understanding that you are drawing two quads which are possibly overlapping. Are they using texture maps? And when you draw the second quad you want to perform a masking operation involving pixels you wrote for the first quad? If you have a quick test image that shows what you are trying to do, that might be helpful in answering your question. Thanks!

Yeah sure. Check this These are two wuads I have drawn. The first has different coordinates. I need to multiply two quad colors.

OK, so it looks like you are trying to simulate a stereoscopic pair of glasses. If you add a second texture to your shader, you should be able to combine them in the shader. So here is some pseudocode you could consider for your shader and your draw code:

So first I'll assume that your input data has at least a position and texture coordinate:

typedef struct
{
    float3 position [[attribute(VertexAttributePosition)]];
    float2 texCoord [[attribute(VertexAttributeTexcoord)]];
} Vertex;

typedef struct
{
    float4 position [[position]];
    float2 texCoord;
} ColorInOut;

typedef struct
{
    matrix_float4x4 projectionMatrix;
    matrix_float4x4 modelViewMatrix;
} Uniforms;

So your vertex shader should be something like:

vertex ColorInOut vertexShader(Vertex in [[stage_in]],
                               constant Uniforms & uniforms [[ buffer(BufferIndexUniforms) ]])
{
    ColorInOut out;
    float4 position = float4(in.position, 1.0);
    out.position = uniforms.projectionMatrix * uniforms.modelViewMatrix * position;
    out.texCoord = in.texCoord;
    return out;
}

In your fragment shader, you will want to add a second input texture:

fragment float4 fragmentShader(ColorInOut in [[stage_in]],
                               constant Uniforms & uniforms [[ buffer(BufferIndexUniforms) ]],
                               texture2d<half> colorMap1     [[ texture(TextureIndexColor1) ]],
                               texture2d<half> colorMap2     [[ texture(TextureIndexColor2) ]])
{
    constexpr sampler colorSampler(mip_filter::linear, mag_filter::linear, min_filter::linear);

    half4 colorSample1  = colorMap1.sample(colorSampler, in.texCoord.xy);
    half4 colorSample2  = colorMap2.sample(colorSampler, in.texCoord.xy);

    // multiply two texture colors together
    half4 colorSample = colorSample1 * colorSample2;

    return float4(colorSample);
}

And in your program, you will specify the second texture:

let TextureIndex1 = 0
let TextureIndex2 = 1
renderEncoder.setFragmentTexture(colorMap1, index: TextureIndex1)
renderEncoder.setFragmentTexture(colorMap2, index: TextureIndex2)

Thanks for your response. But what I did is I have multiple quads and for each quad I have to multiply the color with the mask image(second). So in the first image, the upper. part is differrentt quad, and the lower part is different quad. And If I add mask texture to each quad, it looks like this. I have to mask each quad with. the mask image and multiply specific color component.

Both textures has different vertex coordinates. The first image vertices data is coming form server like -0.026 -0.185 +1.267 -0.108 -0.267 +1.108 +1.026 +1.185 +1.000 topleftX topleft Y toprightX toprightY bottomleftX bottomleftY bottomrightX bottomrightY. And mask image. So i want mask each quad with this mask image.

Perhaps you could post some pseudocode of what you are currently trying to do? Generally masking is either something you do with a stencil buffer, or with alpha blending, or with a shader. As I showed earlier in the shader example, the operation is straightforward. And you will get more performance if you use two textures as input to a shader, rather than reading back from the framebuffer. However, in the screenshots you posted, it's hard to tell what is the source quad + texture map, and what is the masking quad + texture map.

Yea sure. So. this is how I am setting vertex coordinate according to the coordinates coming from server.

vertices = [
      Vertex(position: SIMD3(Float(topLeft.x), Float(topLeft.y), 0), color: SIMD4(1, 0, 0, 1), texture: SIMD2(0, 1)),
      Vertex(position: SIMD3(Float(bottomLeft.x), Float(bottomLeft.y), 0), color: SIMD4(0, 1 , 0, 1), texture: SIMD2(0, 0)),
      Vertex(position: SIMD3(Float(bottomRight.x), Float(bottomRight.y), 0), color: SIMD4(0, 0, 1, 1), texture: SIMD2(1, 0)),
      Vertex(position: SIMD3(Float(topRight.x), Float(topRight.y), 0), color: SIMD4(1, 0, 1, 1), texture: SIMD2(1, 1))
    ]

Now this how I am setting texture,

commandEncoder.setFragmentTexture(texture, index: 0)
commandEncoder.setFragmentTexture(maskTexture, index: 1)

Now for the first texture, I want the image to be setup according. to coordinates. And for second mask texture, it should be full screen texture.

But somehow mask texture is also. taking the same vertex. data (as shown. in above image).

My vertex function.

vertex VertexOut vertex_shader(const VertexIn vertexIn [[ stage_in ]]) {
   
  VertexOut vertexOut;
  vertexOut.position = vertexIn.position;
  vertexOut.color = vertexIn.color;
  vertexOut.textureCoordinates = vertexIn.textureCoordinates;
  vertexOut.points = vertexIn.points;
  return vertexOut;
}

My fragment function,

fragment float4 textured_mask_fragment(VertexOut vertexIn [[stage_in]],
                    texture2d<float> srcTexture [[texture(0)]],
                    texture2d<float> maskTexture [[texture(1)]],
                    sampler sampler2d [[sampler(0)]],
                    constant int &maskComponent [[ buffer(0) ]]) {
   
  float4 srcColor = srcTexture.sample(sampler2d, vertexIn.textureCoordinates);
  float4 maskColor = maskTexture.sample(sampler2d, vertexIn.textureCoordinates);
   
  if (is_null_texture(maskTexture)) {
    return srcColor;
  }
   
  float4 maskedColor;
  if (maskComponent == 3) { // means no mask channel
    maskedColor = srcColor;
  } else {
    maskedColor = srcColor * maskColor[maskComponent];
  }
  return maskedColor;
}.

It's just like for every quad I am adding, I have to multiply the colors with the full screen texture. But the the mask. texture is not coming to. full screen, it is taking same as the vertex data I added for quad.

Since you are sampling with the same set of texture coordinates, then you will get what you are seeing. But since you want to treat the mask as a fullscreen texture, you will need to do one of two things. The first option is to supply a second set of texture coordinates for the mask texture which has been adjusted so that it samples the texture in the right position. The second option is to use your vertexIn.position as the source for your texture coordinates and divide by the screen width and height to map the position to within 0 and 1 to properly sample the texture.

In your above fragment shader, you could try something like:

half2 maskUV = half2(vertexIn.position.xy) / half2(828.0, 1792.0);
float4 maskColor = maskTexture.sample(sampler2d, maskUV);

I'm assuming the width and height is 828 and 1792 based on the images you posted. But you would need to pass these in to your shader so that it works on a variety of devices.

Okay. I will try this.So by this code this will make the mask texture to be in full screen and the previous quad in it’s vertex coordinates according to coming from server.. Right?

In effect yes, because the pixel coordinate of the fragment you are rendering is being used as the coordinates to sample the texture. It's always helpful to note that texture coordinates are always in the range 0 to 1 so that if you rotate the triangle/quad you are rendering, the texture automatically "moves" with the polygon. So if you want to fix the texture so it appears that it is not moving relative to the screen, then you need to assign texture coordinates that correspond to the screen position.

Okay I will try today. I will add answer here after my trial. Please be there. I am very much stuck with this work. Thanks 😄