The loader that segfaults instead of failing

Why this is confusing

Because the obvious mental model is wrong in a reasonable way. MLX runs on the GPU, you want the weights on the GPU, so you load on the GPU stream. That reasoning is fine and the conclusion is not.

Loading is file I/O and memory placement, and on Apple Silicon the memory is unified anyway — there is no transfer to arrange. Loading on the CPU stream does not leave your weights in the wrong place. The computation still happens on the GPU. Only the load changes.

Once you know this it is obvious. Before you know it, you are looking for a memory bug in your own code, because a segmentation fault is what your own bad pointer looks like.

The part that turns a bug into a mystery

mlx_array_eval returns a status code. Ignoring it is the natural thing to do, because most of the time it is zero and checking every call is noise.

Do not ignore it. An unevaluated array does not announce itself. It reads as zeros, silently, with no error anywhere. And once you have zeros flowing through a transformer, the symptom is "the model generates zeros" or "the model outputs garbage" — which sends you looking at the sampling code, or the tokenizer, or the architecture. Everywhere except the loader.

That is the general shape of this whole class of defect, and it is worth stating on its own:

A failure that returns a plausible value is more expensive than a failure that crashes, because the crash tells you where to look.

So: check the return code on every eval. It costs a line each and it converts a class of mysteries into a class of error messages.

How to reproduce it

Load any MLX safetensors file with the default stream, evaluate, and read the first element.

// The version that crashes
mlx_array w;
mlx_load(&w, "model.safetensors", /* stream */ mlx_default_gpu_stream_new());
mlx_array_eval(&w);          // returns 1 — and nobody looks
const float* p;
mlx_array_data_float32(&p, w);
printf("%f\n", p[0]);        // segmentation fault, here

// The version that works
mlx_stream s = mlx_default_cpu_stream_new();
mlx_load(&w, "model.safetensors", s);
if (mlx_array_eval(&w) != 0) { /* now you find out */ }

The crash location is the read, not the load, which is the whole problem: your debugger points at a line that is correct.

What it looks like from Rust

Same thing, one layer up, and the ergonomics make it easier to get wrong: a wrapper that returns Result from the load and swallows the eval will look like it is handling errors properly.

The pattern that holds up is treating evaluation as a fallible operation everywhere, not just where it seems likely to fail. In practice that means a small helper that evaluates and converts the status into your error type, and never calling the raw function directly. It is not interesting code. It is the code that turns a segfault into a sentence.

Why the load path is different at all

Some informed speculation, since this is not documented as such: loading has to place bytes from a file into memory, and the metal path expects to be handed buffers rather than to populate them from a file descriptor. Whatever the internal reason, the observable behaviour is stable and the workaround is trivial.

What matters practically is that this is not a performance decision you are making. You are not choosing to load slowly on the CPU. You are choosing the only path that works, and the computation afterwards is unaffected.

How long this took, and the shape of the search

Two days, and almost all of it spent in the wrong places. The path is worth recording because the wrong places were not stupid ones.

The symptom was garbage output. So the first suspect was sampling — an off-by-one in the top-k selection, a temperature applied twice, a seed doing something unexpected. Sampling is where randomness lives, and randomness is what garbage looks like. Two hours ruling that out by fixing the seed and confirming the same garbage came back identically, which is the useful negative result: deterministic garbage is not a sampling problem.

Then the tokenizer, because garbage that decodes from valid token identifiers looks different from garbage that does not, and this looked like the second. Another few hours confirming the tokenizer round-trips correctly on its own.

Then the architecture, which is where a port is genuinely likely to be wrong: a transposed matrix, a missing normalisation, a rotary embedding applied at the wrong point. This is the expensive one to rule out, because verifying it means comparing intermediate tensors against a reference layer by layer.

And it was during that layer-by-layer comparison that the actual answer appeared, sideways: the first tensor compared was already wrong, before any computation had happened. The weights themselves were zeros.

The general lesson is the one that keeps paying:

Compare against a reference at the earliest point you can, not at the point where the symptom appears.

The symptom appeared at the output, so that is where the search started. The defect was at the input, and half an hour of comparing loaded weights against the reference would have found it on the first day.

The related trap: shards are discovered, not named

While you are in the loader, the neighbouring mistake is worth avoiding too.

A model's weights may be in one file or several, and the count varies by model rather than by convention: Llama-3.2-1B publishes one file, phi-4 publishes two. Code that names model.safetensors works until the first sharded model, and code that assumes model-00001-of-* fails on the first single-file one.

Read the directory, or better, read the weight index — model.safetensors.index.json and its weight_map — which names every shard the model needs. That is also the only reliable way to tell a finished download from an interrupted one, because a file count tells you nothing about whether the right files are present.

The version of this that bit twice

One more thing in the same family, since it cost a second afternoon and has the same shape.

Do not call eval inside the layer loop. There was one there, left over from a proof of concept: eval(&h) after each layer, which is correct and enormously expensive. On a model with sixteen layers that is sixteen GPU synchronisations per token; on one with forty, forty.

The arrays only need to be evaluated at the end of the step. Removing it took generation from 94 to 244 tokens per second — more than half the speed was being spent on synchronising with the GPU for no reason.

It is the same lesson from the other side. The first defect was not checking a result you should have; this one was forcing a result you did not need. Both are about when evaluation happens, and in a lazy graph that is not a detail.

The short version

• Load safetensors on the CPU stream. The computation still runs on the GPU.
• Check the return of every eval. An unevaluated array reads as zeros, silently.
• Discover shards from the index rather than naming files.

Measured against mlx-c on 6 September 2026. If MLX changes this, the date is your warning that this page has not been rechecked since.