MLX Getting Silently Killed Loading a Large Model

Two words, no traceback, no stack, nothing in the Python logs. That absence is the diagnosis: an exception would have printed something, and a signal from the kernel cannot.

What you see

$ python generate.py
zsh: killed     python generate.py

$ echo $?
137

The shell prints that line when a child process is terminated by a signal it did not handle. bash prints Killed instead, and in some contexts you will see Killed: 9. The 137 is the convention for a signalled exit: 128 plus the signal number, and signal 9 is SIGKILL.

Telling a kill from an exception

This is the whole diagnostic question, and it has a clean answer.

• A Python-level failure prints. A MemoryError, an MLX allocation failure, a shape mismatch — all of them raise, unwind the stack, and write a traceback to stderr. If you have a traceback of any kind, you are not in this page’s case.
• SIGKILL cannot be caught. No handler runs, no finally block executes, no atexit hook fires, nothing is flushed. The process stops between one instruction and the next. Silence is the signature.
• The exit code is definitive. echo $? immediately after. 137 is SIGKILL; 139 is a segmentation fault, which is a different problem; a small number is an ordinary Python exit.

To confirm that memory pressure was the reason rather than something else sending the signal, macOS records the termination. Jetsam events are written to the diagnostic reports directory, and the unified log carries the decision:

ls -t ~/Library/Logs/DiagnosticReports | head
log show --last 10m --predicate 'eventMessage CONTAINS "jetsam"' | tail -20

A JetsamEvent report naming your Python process, timed to the second the shell printed, closes the question.

Sizing the model against unified memory

Apple Silicon shares one pool of memory between CPU and GPU, so “does it fit” is one sum rather than two. You can do that sum before downloading anything.

Weights first. A model quantized to 4 bits in MLX carries a scale and a bias per group alongside the packed weights, so the effective rate is above four bits per parameter — call it about 4.5 for a group-of-64 layout, which is the assumption everything below rests on. For a 70B model:

weights = 70e9 params x 4.5 bits / 8 bits per byte
        = 39.4e9 bytes
        ~ 39 GB   (before any context at all)

Then the KV cache, which is the part people forget and the reason a model that loaded yesterday dies today after a long conversation. The per-token cost is 2 × layers × kv_heads × head_dim × bytes, computable from the model’s own configuration file — the worked example is in the slot-budget page, and the number is measured in tens of kilobytes per token, so a long context costs gigabytes.

Add the operating system and whatever else is open. On a 48 GB machine, a 39 GB model plus a growing cache plus a browser is not a comfortable fit, and the kill usually arrives partway through loading rather than at the start — which is why it looks like a loader bug.

One more thing the arithmetic explains: why the kill so often happens partway through loading rather than at the first line. Weights are read from disk and materialised progressively, so memory climbs steadily for the duration of the load. The process is terminated at the moment the climb crosses the threshold, which is usually well into the file — which is why the symptom reads as “it almost worked” and invites another attempt rather than a smaller model.

Wired memory, and why raising the limit is risky

MLX asks Metal how much memory the GPU may hold as its working set, and this is where the second failure mode lives. Memory wired for the GPU cannot be paged out. That is what makes it fast, and it is also why exceeding the budget produces a kill rather than a slowdown: there is no swap path for wired pages, so the memory manager’s only remaining option is to terminate something.

There is a well-known family of advice about raising the wired limit with a sysctl to let a larger model fit. Treat it carefully. The mlx-lm tracker carries a report of unbounded KV growth against a raised wired limit producing a kernel panic rather than a process termination — because once the memory is wired, the OOM killer cannot reclaim it, and the failure moves down into the GPU driver. A crashed process is recoverable; a panicked machine is not, and it is not a good trade for a few gigabytes.

Making it fit

1. Confirm it was a kill: echo $? gives 137, and there is no traceback. If either is untrue, stop and read the actual error.
2. Add up the on-disk size of the weights and compare it to your installed memory. If weights alone exceed roughly two thirds of RAM, no configuration change will save it.
3. Bound the KV cache. mlx_lm.generate accepts a maximum cache size, and capping it converts an unbounded growth that eventually kills the process into a bounded one that does not. This is the single highest-value change for long conversations.
4. Step down a quantization level, or a model size. 4-bit rather than 8-bit is close to half the weights; the quality trade is discussed in quantization at inference.
5. Close what else is resident before deciding it does not fit. A browser with many tabs is measured in gigabytes and is frequently the difference.