I'm attempting to get the Xcode model preview to work with instance segmentation models, specifically RegNetX Mask R-CNN. I've already tried using the imageSegmenter model preview type and setting it in my Core ML model specification. However, this crashes the preview when attempting to load the model. I believe this might be due to a slightly different output format. The model I'm using gives three outputs:
- Detections with bounding boxes, (x1, y1, x2, y2, confidence)
- Labels, the index of the detected label
- Masks, 28×28 image, which should be resized to the dimensions of the bounding boxes and fit on the x1, y1 position
I'm not sure how the Xcode preview handles different output formats, or if it only supports one specific format.
Ultimately, my goal in this effort is to use the Vision framework for instance segmentation models. I'm not sure if this is currently supported, as I am unable to find proper information on existing solutions. I'm also unsure which VN object would be necessary for this, if it even exists. Furthermore, I've looked at VNDetectedObjectObservation. This object provides a segmentation mask, but I'm not sure if that would work with the output format of my model, since it has to upscale and fit the mask on the bounding box as well.
