CoreML Conversion Display Issues

Hello!

I have a TrackNet model that I have converted to CoreML (.mlpackage) using coremltools, and the conversion process appears to go smoothly as I get the .mlpackage file I am looking for with the weights and model.mlmodel file in the folder. However, when I drag it into Xcode, it just shows up as 4 script tags instead of the model "interface" that is typically expected. I initially was concerned that my model was not compatible with CoreML, but upon logging the conversions, everything seems to be converted properly.

I have some code that may be relevant in debugging this issue: How I use the model:

model = BallTrackerNet() # this is the model architecture which will be referenced later
device = self.device # cpu
model.load_state_dict(torch.load("models/balltrackerbest.pt", map_location=device)) # balltrackerbest is the weights
model = model.to(device)
model.eval()

Here is the BallTrackerNet() model itself

import torch.nn as nn
import torch

class ConvBlock(nn.Module):
    def __init__(self, in_channels, out_channels, kernel_size=3, pad=1, stride=1, bias=True):
        super().__init__()
        self.block = nn.Sequential(
            nn.Conv2d(in_channels, out_channels, kernel_size, stride=stride, padding=pad, bias=bias),
            nn.ReLU(),
            nn.BatchNorm2d(out_channels)
        )

    def forward(self, x):
        return self.block(x)
    
class BallTrackerNet(nn.Module):
    def __init__(self, out_channels=256):
        super().__init__()
        self.out_channels = out_channels

        self.conv1 = ConvBlock(in_channels=9, out_channels=64)
        self.conv2 = ConvBlock(in_channels=64, out_channels=64)
        self.pool1 = nn.MaxPool2d(kernel_size=2, stride=2)
        self.conv3 = ConvBlock(in_channels=64, out_channels=128)
        self.conv4 = ConvBlock(in_channels=128, out_channels=128)
        self.pool2 = nn.MaxPool2d(kernel_size=2, stride=2)
        self.conv5 = ConvBlock(in_channels=128, out_channels=256)
        self.conv6 = ConvBlock(in_channels=256, out_channels=256)
        self.conv7 = ConvBlock(in_channels=256, out_channels=256)
        self.pool3 = nn.MaxPool2d(kernel_size=2, stride=2)
        self.conv8 = ConvBlock(in_channels=256, out_channels=512)
        self.conv9 = ConvBlock(in_channels=512, out_channels=512)
        self.conv10 = ConvBlock(in_channels=512, out_channels=512)
        self.ups1 = nn.Upsample(scale_factor=2)
        self.conv11 = ConvBlock(in_channels=512, out_channels=256)
        self.conv12 = ConvBlock(in_channels=256, out_channels=256)
        self.conv13 = ConvBlock(in_channels=256, out_channels=256)
        self.ups2 = nn.Upsample(scale_factor=2)
        self.conv14 = ConvBlock(in_channels=256, out_channels=128)
        self.conv15 = ConvBlock(in_channels=128, out_channels=128)
        self.ups3 = nn.Upsample(scale_factor=2)
        self.conv16 = ConvBlock(in_channels=128, out_channels=64)
        self.conv17 = ConvBlock(in_channels=64, out_channels=64)
        self.conv18 = ConvBlock(in_channels=64, out_channels=self.out_channels)

        self.softmax = nn.Softmax(dim=1)
        self._init_weights()
                  
    def forward(self, x, testing=False): 
        batch_size = x.size(0)
        x = self.conv1(x)
        x = self.conv2(x)    
        x = self.pool1(x)
        x = self.conv3(x)
        x = self.conv4(x)
        x = self.pool2(x)
        x = self.conv5(x)
        x = self.conv6(x)
        x = self.conv7(x)
        x = self.pool3(x)
        x = self.conv8(x)
        x = self.conv9(x)
        x = self.conv10(x)
        x = self.ups1(x)
        x = self.conv11(x)
        x = self.conv12(x)
        x = self.conv13(x)
        x = self.ups2(x)
        x = self.conv14(x)
        x = self.conv15(x)
        x = self.ups3(x)
        x = self.conv16(x)
        x = self.conv17(x)
        x = self.conv18(x)
        # x = self.softmax(x)
        out = x.reshape(batch_size, self.out_channels, -1)
        if testing:
            out = self.softmax(out)
        return out                       
    
    def _init_weights(self):
        for module in self.modules():
            if isinstance(module, nn.Conv2d):
                nn.init.uniform_(module.weight, -0.05, 0.05)
                if module.bias is not None:
                    nn.init.constant_(module.bias, 0)

            elif isinstance(module, nn.BatchNorm2d):
                nn.init.constant_(module.weight, 1)
                nn.init.constant_(module.bias, 0)

I have been struggling with this conversion for almost 2 weeks now so any help, ideas or pointers would be greatly appreciated!

Thanks!

Michael

Answered by Frameworks Engineer in 822551022

So, somewhere between coremltools and Xcode, BallTracker.mlpackage got replaced with some text garbage. The attached file in the feedback assistant was also broken, as I showed above.

It can be some file operation user error, a bug in APFS file system (unlikely), or misbehaving disk drive (very unlikely). Sorry, I have honestly no idea. It seems to me something outside of CoreML framework.

You can verify whether your Xcode project has a broken one or a good one by:

xcrun coremlcompiler metadata $(find /path/to/your/Xcode/project/ -name "BallTracker.mlpackage")

Additionally, in case needed as well, here is my conversion script:

import torch
import coremltools as ct
import numpy as np
import logging
from ball_tracker_model import BallTrackerNet

def convert_to_coreml(model_path):
    logging.basicConfig(level=logging.DEBUG)

    model = BallTrackerNet()
    model.load_state_dict(torch.load(model_path, map_location='cpu'))
    model.eval()
    
    example_input = torch.rand(1, 9, 360, 640)
    
    # Trace the model to verify shapes
    traced_model = torch.jit.trace(model, example_input)
    
    model_coreml = ct.convert(
        traced_model,
        inputs=[
            ct.TensorType(
                name="input_frames",
                shape=(1, 9, 360, 640),
                dtype=np.float32,
            )
        ],
        convert_to="mlprogram",
        minimum_deployment_target=ct.target.iOS15,
    )
    
    model_coreml.save("BallTracker.mlpackage")
    return model_coreml

# Run conversion
try:
    model = convert_to_coreml("balltrackerbest.pt")
    print("Conversion successful!")
except Exception as e:
    print(f"Conversion error: {str(e)}")

Thanks again!

Hi @michaeldegoat!

However, when I drag it into Xcode, it just shows up as 4 script tags instead of the model "interface" that is typically expected.

I'm not sure I understand what 4 script tags mean. Maybe could you attach a screenshot?

Also, what does this command say?

xcrun coremlcompiler metadata path/to/model.mlpackage

Yes, this is what I am seeing in Xcode.

xcrun coremlcompiler metadata path/to/model.mlpackage says the following:

[
  {
    "metadataOutputVersion" : "3.0",
    "storagePrecision" : "Float16",
    "outputSchema" : [
      {
        "hasShapeFlexibility" : "0",
        "isOptional" : "0",
        "dataType" : "Float32",
        "formattedType" : "MultiArray (Float32 1 × 256 × 230400)",
        "shortDescription" : "",
        "shape" : "[1, 256, 230400]",
        "name" : "var_462",
        "type" : "MultiArray"
      }
    ],
    "modelParameters" : [

    ],
    "specificationVersion" : 6,
    "mlProgramOperationTypeHistogram" : {
      "Cast" : 2,
      "Conv" : 18,
      "Relu" : 18,
      "BatchNorm" : 18,
      "Reshape" : 1,
      "UpsampleNearestNeighbor" : 3,
      "MaxPool" : 3
    },
    "computePrecision" : "Mixed (Float16, Float32, Int32)",
    "isUpdatable" : "0",
    "availability" : {
      "macOS" : "12.0",
      "tvOS" : "15.0",
      "visionOS" : "1.0",
      "watchOS" : "8.0",
      "iOS" : "15.0",
      "macCatalyst" : "15.0"
    },
    "modelType" : {
      "name" : "MLModelType_mlProgram"
    },
    "userDefinedMetadata" : {
      "com.github.apple.coremltools.source_dialect" : "TorchScript",
      "com.github.apple.coremltools.source" : "torch==2.5.1",
      "com.github.apple.coremltools.version" : "8.1"
    },
    "inputSchema" : [
      {
        "hasShapeFlexibility" : "0",
        "isOptional" : "0",
        "dataType" : "Float32",
        "formattedType" : "MultiArray (Float32 1 × 9 × 360 × 640)",
        "shortDescription" : "",
        "shape" : "[1, 9, 360, 640]",
        "name" : "input_frames",
        "type" : "MultiArray"
      }
    ],
    "generatedClassName" : "BallTracker",
    "method" : "predict"
  }
]

Does the issue happen only with this particular BallTracker.mlpackage? Then, could you file a feedback assistant report with the model attached?

Or, does it happen with other .mlpackage bundles? Then the file type registration could be messed up. Does the command below show anything interesting?

/System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Versions/Current/Support/lsregister -dump | grep -B 10 "tags:[[:space:]]*\.mlpackage"

Yes this only happens with BallTracker.mlpackage. Yes, I have filed a feedback assistant report and attached the model.

@michaeldegoat, do you have the feedback number so that I can look it up?

Yes, this is the link: https://feedbackassistant.apple.com/feedback/16382305 and the feedback number I believe is FB16382305 Thanks!

Oh interesting. BallTracker_42137490.mlpackage is literally a text file with the contents:

(base) ➜  Downloads hexdump -C BallTracker_42137490.mlpackage
00000000  3c 73 63 72 69 70 74 3e  73 74 61 72 74 28 22 2f  |<script>start("/|
00000010  55 73 65 72 73 2f 6d 69  63 68 61 65 6c 7a 68 61  |Users/michaelzha|
00000020  6e 67 2f 44 65 73 6b 74  6f 70 2f 43 6f 6e 76 65  |ng/Desktop/Conve|
00000030  72 74 65 72 73 2f 42 61  6c 6c 54 72 61 63 6b 65  |rters/BallTracke|
00000040  72 2e 6d 6c 70 61 63 6b  61 67 65 2f 22 29 3b 3c  |r.mlpackage/");<|
00000050  2f 73 63 72 69 70 74 3e  0a 3c 73 63 72 69 70 74  |/script>.<script|
00000060  3e 6f 6e 48 61 73 50 61  72 65 6e 74 44 69 72 65  |>onHasParentDire|
00000070  63 74 6f 72 79 28 29 3b  3c 2f 73 63 72 69 70 74  |ctory();</script|
00000080  3e 0a 3c 73 63 72 69 70  74 3e 61 64 64 52 6f 77  |>.<script>addRow|
00000090  28 22 44 61 74 61 22 2c  22 44 61 74 61 22 2c 31  |("Data","Data",1|
000000a0  2c 39 36 2c 22 39 36 20  42 22 2c 31 37 33 37 34  |,96,"96 B",17374|
000000b0  37 32 36 36 39 2c 22 31  2f 32 31 2f 32 35 2c 20  |72669,"1/21/25, |
000000c0  31 30 3a 31 37 3a 34 39  e2 80 af 41 4d 22 29 3b  |10:17:49...AM");|
000000d0  3c 2f 73 63 72 69 70 74  3e 0a 3c 73 63 72 69 70  |</script>.<scrip|
000000e0  74 3e 61 64 64 52 6f 77  28 22 4d 61 6e 69 66 65  |t>addRow("Manife|
000000f0  73 74 2e 6a 73 6f 6e 22  2c 22 4d 61 6e 69 66 65  |st.json","Manife|
00000100  73 74 2e 6a 73 6f 6e 22  2c 30 2c 36 31 37 2c 22  |st.json",0,617,"|
00000110  36 31 37 20 42 22 2c 31  37 33 37 34 37 32 36 37  |617 B",173747267|
00000120  30 2c 22 31 2f 32 31 2f  32 35 2c 20 31 30 3a 31  |0,"1/21/25, 10:1|
00000130  37 3a 35 30 e2 80 af 41  4d 22 29 3b 3c 2f 73 63  |7:50...AM");</sc|
00000140  72 69 70 74 3e 0a                                 |ript>.|
00000146

The correct .mlpackage is supposed to be a bundle (= directory.) So, this file is just broken.

coremlcompiler metadata command does not work either.

(base) ➜  Downloads xcrun coremlcompiler metadata BallTracker_42137490.mlpackage
coremlcompiler: error: unable to read document: /Users/mito/Downloads/BallTracker_42137490.mlpackage
detail: validator error: Model specification version field missing or corrupt.
(null)%       

I would suggest to remove the current BallTracker_42137490.mlpackage from the Xcode project and then add the one that works correctly with coremlcompiler metadata command.

Oh, that is interesting. The BallTracker.mlpackage file I attached to the feedback assistant was the one I ran the xcrun coremlcompiler metadata path/to/model.mlpackage command on. If this produces:

[
  {
    "metadataOutputVersion" : "3.0",
    "storagePrecision" : "Float16",
    "outputSchema" : [
      {
        "hasShapeFlexibility" : "0",
        "isOptional" : "0",
        "dataType" : "Float32",
        "formattedType" : "MultiArray (Float32 1 × 256 × 230400)",
        "shortDescription" : "",
        "shape" : "[1, 256, 230400]",
        "name" : "var_462",
        "type" : "MultiArray"
      }
    ],
    "modelParameters" : [

    ],
    "specificationVersion" : 6,
    "mlProgramOperationTypeHistogram" : {
      "Cast" : 2,
      "Conv" : 18,
      "Relu" : 18,
      "BatchNorm" : 18,
      "Reshape" : 1,
      "UpsampleNearestNeighbor" : 3,
      "MaxPool" : 3
    },
    "computePrecision" : "Mixed (Float16, Float32, Int32)",
    "isUpdatable" : "0",
    "availability" : {
      "macOS" : "12.0",
      "tvOS" : "15.0",
      "visionOS" : "1.0",
      "watchOS" : "8.0",
      "iOS" : "15.0",
      "macCatalyst" : "15.0"
    },
    "modelType" : {
      "name" : "MLModelType_mlProgram"
    },
    "userDefinedMetadata" : {
      "com.github.apple.coremltools.source_dialect" : "TorchScript",
      "com.github.apple.coremltools.source" : "torch==2.5.1",
      "com.github.apple.coremltools.version" : "8.1"
    },
    "inputSchema" : [
      {
        "hasShapeFlexibility" : "0",
        "isOptional" : "0",
        "dataType" : "Float32",
        "formattedType" : "MultiArray (Float32 1 × 9 × 360 × 640)",
        "shortDescription" : "",
        "shape" : "[1, 9, 360, 640]",
        "name" : "input_frames",
        "type" : "MultiArray"
      }
    ],
    "generatedClassName" : "BallTracker",
    "method" : "predict"
  }
]

Then in theory it should work in Xcode? Unfortunately this is not the case for my model though. Does my conversion code look off in any places? Do you have any other recommendations for getting the model working on Xcode?

Sorry for the long thread, just haven't seen anyone else online have a similar issue and want to get to the bottom of this.

Thanks!

Additionally, when I run the model with test data in VSCode, I am getting an expected output so I think the conversion itself is working, the model is just not getting properly read by Xcode for some reason:

model = ct.models.MLModel("BallTracker.mlpackage")
input_data = np.random.rand(1, 9, 360, 640).astype(np.float32)
inputs = {"input_frames": input_data}

# Predict with the model
results = model.predict(inputs)
print(results) # this yields valid results
Accepted Answer

So, somewhere between coremltools and Xcode, BallTracker.mlpackage got replaced with some text garbage. The attached file in the feedback assistant was also broken, as I showed above.

It can be some file operation user error, a bug in APFS file system (unlikely), or misbehaving disk drive (very unlikely). Sorry, I have honestly no idea. It seems to me something outside of CoreML framework.

You can verify whether your Xcode project has a broken one or a good one by:

xcrun coremlcompiler metadata $(find /path/to/your/Xcode/project/ -name "BallTracker.mlpackage")

I found out what it was. I can't drag the model directly from VSCode to Xcode so I first drag the file onto my desktop. This makes it show up as a txt for whatever reason so I had to import the file by adding the file from Xcode, and referencing the file in VSCode directly.

Thank you for the update. I am glad you resolved the issue!

CoreML Conversion Display Issues
 
 
Q