“MLX”

Performance wise what are trade-offs when running an MLX-backed model on-device compared to using the system's AFM Core model? Also semiconnected: How do I use the 'model judge evaluator' to compare the accuracy of a custom LoRA adapter against the system's private cloud compute models?

Hi all, I'm trying to find out if/when we can expect mxfp8/mxfp4 support on Apple Silicon. I've noticed that mlx now has casting data types, but all computation is still done in bf16. Would be great to reduce power consumption with support for these lower precision data types since edge inference is already typically done at a lower precision! Thanks in advance.

Metal offers both threadgroup_barrier() and simdgroup_barrier(). I understand the need for threadroup barriers — it would not be possible to rely on well cooperation between threads in a threadgroup without them, as different threads can execute on different SIMD partitions at different times. But I don't really get the simdgroup_barrier() — it was my impression that all threads in a simdgroup execute in lockstep and this if one thread in a simdgroup makes progress, all other active threads in the simdgroup are also guaranteed to make progress. If this were not the case we'd need to insert simdgroup barrier pretty much any time we read or write any storage or perform SIMD-scoped operations. It doesn't seem like Apple uses simdgroup_barrier() in any of their sample code. In fact, it seems like it's a no-op on current Apple Silicon hardware. Is there a situation when I need to use simdgroup barriers or is this a superfluous operation? P.S. It seems that Apple engineers are as confused by this as I am, see https

I just watched the October 30 MacBook Pro Announcement where they talked about on-device local LLMs for the M4s. What developer training resources are available, where we can learn how to use custom llm models and build our Swift apps to use both Apple Intelligence and other llm models on device? Is the guidance to follow MLX github repos, or were those experimental and now there is an approved workflow and tooling? https://www.youtube.com/watch?v=G0cmfY7qdmY

LLVM Linker Crash on ARM64 with bfloat16 Symbols (Xcode 17.0.0) We're encountering a critical linker crash in Xcode 17.0.0 (clang-1700.4.4.1) on macOS 15.1.0 (Darwin 25.1.0) with Apple Silicon M3 Max when linking a pybind11 C++ extension against the MLX framework (v0.30.1). The linker consistently crashes with LLVM ERROR: No way to correctly truncate anything but float to bfloat during the linking phase, even though our code uses only integer types (int64, uint32) for BPE tokenization and never directly references bfloat16 types. Error Details: [100%] Linking CXX shared module _metal_trainer.cpython-312-darwin.so LLVM ERROR: No way to correctly truncate anything but float to bfloat clang++: error: unable to execute command: Abort trap: 6 clang++: error: linker command failed due to signal (use -v to see invocation) Reproduction: Install MLX framework: pip install mlx (any version with bfloat16 support) Create a minimal pybind11 extension that links against MLX: Compiler: Ap

I can't use cloud-based AI providers, so I'm very excited about the potential of local models with Xcode's Coding Assistant. I've been playing around with a local MLX model, and have been generally pretty satisfied with it. But I'm curious - what features am I missing out on by using a local model instead of a larger, cloud-based model? Is Xcode's ability to run tests, build the project, etc. limited by the use of a local model? Does it limit the ability to do more complex agentic workflows? If so, how? Basically, local models in Xcode seem to be working pretty well for me, and I'm curious to know what cloud-based models can offer that I don't already have.

I ran into a gap in the MLXVLM → MLXLLM factory fallback and wanted to check whether this is expected behavior or something worth filing. Context: I maintain a Swift-native macOS agent built on MLX Swift, with a hardware-adaptive catalog of local models. I tried adding mlx-community/Qwen3.5-9B-4bit as a plain text model. Its config.json ships a full vision_config block — it looks like a multimodal config — but the actual checkpoint has no vision tower weights at all. It's genuinely text-only; the metadata just doesn't reflect that. ModelFactoryRegistry tries MLXVLM before falling back to MLXLLM. I expected that to fail cleanly and fall through, since there's no vision encoder to load. It doesn't. MLXVLM's Qwen35Configuration declares visionConfiguration as a non-optional property mapped to vision_config. Since the key is present in the raw config, decoding succeeds — there's nothing to catch at that stage. So MLXVLM proceeds to actually load the model as a VLM, and it crashes later, inside W

At WWDC25 we launched a new type of Lab event for the developer community - Group Labs. A Group Lab is a panel Q&A designed for a large audience of developers. Group Labs are a unique opportunity for the community to submit questions directly to a panel of Apple engineers and designers. Here are the highlights from the WWDC25 Group Lab for Machine Learning and AI Frameworks. What are you most excited about in the Foundation Models framework? The Foundation Models framework provides access to an on-device Large Language Model (LLM), enabling entirely on-device processing for intelligent features. This allows you to build features such as personalized search suggestions and dynamic NPC generation in games. The combination of guided generation and streaming capabilities is particularly exciting for creating delightful animations and features with reliable output. The seamless integration with SwiftUI and the new design material Liquid Glass is also a major advantage. When should I still bring my own LLM via

After running performance test on my CoreML qwen3 vision, I appreciated the update where results were viewable... ON Mac it mentions Ios18 and im not sure if or how to change.. that bottle neck lead to rebuilding CoreML view. I woke up and realized I have all the pieces together... and ended up with a swift package working demo of Clawbot.. the current issue is Im trying to use gguf 3b to code it.. I have become well aware that everything I create using the big models, they soon become the default themes /layouts for everyone else simply asking for this or that (I appoligise) so here I am asking (while looking to schedule meet with dev) if its possible to speak with anyone about th 1000s of Apple Intelligence PCC, Xcode, and vision reports and feedback ive sent , in terms of just general ways I can work more efficiently without the crash... ive already build a TUI for MLX but the tools for coreML while seems promising are not intuitive, but the vision format instruction was nice to see. Anyway my que

Hi, I am looking for a routine to perform complex-valued linear algebra on the GPU in python for scientific programming, in particular quantum physics simulations. At the moment I am looking for a routine for complex-valued matrix multiplication. I found MLX has a routine for float matrix multiplication, but it does not directly work for complex-valued matrices. I figured a work-around by splitting the complex valued matrix into real and imaginary part and working with the pair, but it makes it cumbersome to integrate with the remainder of the code. I was hoping for a library-based implementation similar to cupy. I also tried out using the tensorflow linear algebra routines, but I couldn't get them to run on the GPU by now. Specifically, a testfile with a tensorflow.keras.applications.ResNet50 routine runs on the GPU, but the routines from tensorflow.linalg and tensorflow.math that I tested (matmul, expm, eigh) were not running on the GPU. Any advice on how to make linear algebra calculations on mac

I have used mlx_lm.lora to fine tune a mistral-7b-v0.3-4bit model with my data. I fused the mistral model with my adapters and upload the fused model to my directory on huggingface. I was able to use mlx_lm.generate to use the fused model in Terminal. However, I don't know how to load the model in Swift. I've used Imports import SwiftUI import MLX import MLXLMCommon import MLXLLM let modelFactory = LLMModelFactory.shared let configuration = ModelConfiguration( id: pharmpk/pk-mistral-7b-v0.3-4bit ) // Load the model off the main actor, then assign on the main actor let loaded = try await modelFactory.loadContainer(configuration: configuration) { progress in print(Downloading progress: (progress.fractionCompleted * 100)%) } await MainActor.run { self.model = loaded } I'm getting an error runModel error: downloadError(A server with the specified hostname could not be found.) Any suggestions? Thanks, David PS, I can load the model from the app bundle // directory: Bundle.main.resourceURL! but it's too bi

Hi all :) I tried to raise this in the group lab and was pointed here. I’m seeing a flat per-app memory ceiling of about 6 GB on iPhone, even on devices with more physical RAM and with com.apple.developer.kernel.increased-memory-limit. Measured with os_proc_available_memory() plus task_vm_info.phys_footprint, the total process budget stays around 6144 MB on both: iPhone 16 Pro Max, 8 GB RAM iPhone 17 Pro Max, 12 GB RAM This came up while running Gemma 4 multimodal support in mlx-swift-lm (PR #343). The model loads at about 4.4 GB resident, leaving roughly 1.7 GB for inference/prefill. Reducing a GPU buffer cache from 512 MB to 64 MB recovered enough headroom to avoid jetsam and allowed a full image + video + audio multimodal test to complete, so the measurement seems to reflect a real per-process limit rather than free system memory. I re-measured the ceiling on the 12 GB phone with these capabilities: increased-memory-limit only: ~6144 MB increased-memory-limit + extended-virtual-addressing: ~6144 M

Hi everyone, I've been working on a problem that I think many of us building AI in Swift are facing: how do you debug why an AI agent behaves differently between runs? Traditional logging tells you what happened. But when a model skips a step, changes its reasoning order, or produces a different output with identical input — you're left staring at walls of logs with no clear answer to why. I've released DProvenanceKit — a reasoning observability framework for Swift that lets you: Record every reasoning step an agent takes (non-blocking, async-safe) Query for reasoning patterns (find runs where X happened but Y didn't) Diff two executions to see structural differences Detect regressions automatically with rule-based validation Think of it as Git for AI logic. Example: Swift // Record an execution try await DProvenanceKit.run(contextID: case-123, store: store) { DProvenanceKit.record(.documentEvaluated(documentID: DocA, score: 0.95)) DProvenanceKit.record(.conflictDetected(reason: timeline_inconsistency)) DProv