A default that is inverted between two files

What the flag actually controls

A language model has an embedding table that turns token identifiers into vectors at the input, and a projection that turns vectors back into scores over the vocabulary at the output. Those two are the same shape, transposed.

Tied means they are the same weights: the output projection is the transpose of the input embedding, and the file contains one matrix. Untied means they are separate, learned independently, and the file contains two — the second usually called lm_head.weight.

Tying saves a large amount of memory on a small model, where the vocabulary matrix can be a substantial share of the parameters. Untying gives the model more capacity. Different families made different choices, which is why the flag exists.

Why getting it wrong is quiet

If you assume tied when the model is untied, you use the input embedding as the output projection. The matrix has the right shape. The multiplication succeeds. You get logits.

They are logits computed with the wrong matrix, so the model's output distribution is not the one it was trained to produce — but it is not noise either, because the two matrices are related in a model that was trained with them separate. The text stays fluent. It just gets worse in a way that reads as "this quantisation is a bit lossy" or "this model is not very good".

The reverse direction is louder: assuming untied when the weights are tied means looking for lm_head.weight and not finding it, which at least fails.

The fix is to stop guessing

The reliable answer is not a better default. It is to look at the weights.

If lm_head.weight is present in the safetensors, the embeddings are untied — because an untied model has to ship that matrix, and a tied model has no reason to. If it is absent, they are tied.

That check is not a heuristic. It is the fact itself: the presence of the tensor is what untied means at the file level. A config field is a claim about the file, and where the two disagree the file wins.

So the order is: read the config field if it exists, and if it does not, look for the tensor. Never fall back to a constant.

Checking a model in thirty seconds

Since the answer is in the weights, here is the practical recipe, which does not require downloading anything.

Open the model on Hugging Face and look at the file list. If you see lm_head.weight in a safetensors index, or a file whose name suggests it, the embeddings are untied. On a sharded model, the index file — model.safetensors.index.json — has a weight_map listing every tensor name, and it is a plain JSON file you can read in the browser.

Then open config.json and look for tie_word_embeddings. Three outcomes:

• It says true and there is no lm_head.weight — consistent, tied, nothing to think about.
• It says false and lm_head.weight exists — consistent, untied, likewise.
• It says nothing — this is the common case, and the one where the two libraries disagree. The weights decide.

The fourth combination, where the config and the weights contradict each other, is rare and worth treating as a broken repository rather than a puzzle to solve.

The general pattern

This is one instance of something worth recognising, because it repeats across every model runtime.

Reference implementations are written per family, and their defaults are local knowledge. A value in llama.py is not a statement about language models; it is a statement about llama. Reading one file and generalising from it is how you inherit an assumption that was never intended to travel.

The same shape shows up elsewhere in the same codebase. Attention biases, for example: qwen2 has them on the query, key and value projections and declares them nowhere, because in mlx-lm they live in the architecture rather than the config. Llama makes them optional through attention_bias. A port that reads only the config gets qwen2 wrong; a port that always adds biases gets llama wrong. Looking at whether the tensor exists answers both.

Which is the same rule as before, and it generalises:

When the configuration and the weights disagree about what a model is, the weights are the model.

How to catch it before shipping

The only reason this was found rather than shipped is that correctness here is checked against a reference rather than by reading the output.

mlx-lm generates oracles: for a given model and prompt, the exact token sequence and the raw logits. A port has to reproduce them. A wrong output projection changes the logits immediately and visibly — in numbers, not in vibes — and the test fails on the first token.

If you are porting architectures and you do not have that, it is worth building before the second family rather than after the tenth. The alternative is reading generated text and forming an impression, and an impression cannot distinguish "wrong matrix" from "small model".

A caveat on comparing logits, since it is the same afternoon's work: the tolerance cannot go below one step of the logit type. Qwen3 produces logits in bfloat16, where at a magnitude of 16 one step is 0.125 — so demanding 0.05 is demanding bit-for-bit equality while calling it a tolerance.

What it costs when it is right

Worth a paragraph, because the flag is not free either way and the trade is real.

On a small model the vocabulary matrix is a large share of everything. A model with a 128,000-token vocabulary and a hidden size of 2,048 carries about 262 million parameters in that matrix alone — which on a one-billion-parameter model is roughly a quarter of it. Tying halves that cost, and on a model that small the memory saved is the difference between fitting comfortably and not.

On a large model the same matrix is a rounding error, and the extra capacity from untying is worth more than the memory. Which is why the choice correlates with size, and why a default that was right for one family is wrong for the other — the two libraries were not being careless. They were each describing their own models.

The short version

• tie_word_embeddings defaults to false in phi3 and true in llama, in the same library.
• Getting it wrong produces fluent, worse text, not an error.
• Do not pick a default: check whether lm_head.weight is in the weights.
• Same rule for attention biases, and for anything else the config can omit.

One last thing, since it is the question this always raises: no, you cannot tell from the model's name, its parameter count, or its family. Two models from the same publisher, released a month apart, can differ. The file list is the only place the answer lives.

Checked against mlx-lm 0.31.3 on 6 September 2026.