Running Larger Local Models on a Jetson AGX Orin

The AGX Orin is the edge module where a 70B model stops being absurd and starts being merely slow. Both halves of that sentence come out of two numbers on NVIDIA’s datasheet.

The two configurations

From the NVIDIA Jetson AGX Orin Series module datasheet (DS-10662) and NVIDIA’s Jetson AGX Orin Series technical brief, the two production modules are:

• AGX Orin 32GB — 200 TOPS INT8, 1792 CUDA cores and 56 Tensor cores, 8-core Arm Cortex-A78AE, 32 GB of 256-bit LPDDR5 at 204.8 GB/s, 15–40 W.
• AGX Orin 64GB — 275 TOPS INT8, 2048 CUDA cores and 64 Tensor cores, 12-core Cortex-A78AE, 64 GB of 256-bit LPDDR5 at 204.8 GB/s, 15–60 W.

The detail worth pausing on is that the bandwidth figure is identical. NVIDIA’s technical brief derives it the obvious way: LPDDR5 at 3200 MHz, 6400 Gbps per pin, across a 256-bit interface. Both modules have the same 256-bit bus at the same clock, so the 64 GB part gives you twice the capacity at exactly the same rate — which means every model that only fits on the 64 GB part runs proportionally slower than the smaller one it replaced. Buying capacity here does not buy speed.

Unified memory is not all yours

There is no separate VRAM on a Jetson. The GPU and the CPU share one LPDDR5 pool, which is why a 64 GB module competes with discrete cards that cost more — and also why the usable figure is well under the nameplate. Off the top come the Linux kernel and userspace, the display stack if one is attached, camera and codec buffers, and whatever else the device is actually for. On a headless JetPack install, budgeting 4–6 GB for everything that is not your model is realistic and leaves room for the allocator.

The upside of unified memory is that there is no host-to-device copy for weights and no PCIe transfer in the decode loop. The downside is that your model, your application and the OS are all drawing from the same 204.8 GB/s. A pre-processing thread hammering memory does not just compete for CPU; it competes for the exact resource the decode loop is bounded by.

The size ceiling, derived

Using the same published-file-size method as elsewhere in this cluster — bartowski’s Llama-3.2-1B-Instruct-GGUF card lists F16 at 2.48 GB and Q4_K_M at 0.81 GB, a ratio of 0.327, i.e. about 5.2 bits per weight:

weight bytes ~= parameters x 5.2 / 8      (Q4_K_M)

 13B ->  8.5 GB
 34B -> 22.1 GB
 70B -> 45.5 GB
123B -> 80.0 GB

Against a usable budget of roughly 27 GB on the 32 GB module and 58 GB on the 64 GB module:

• 32 GB module — 34B at Q4_K_M with about 5 GB left for context and runtime. A 70B does not fit at 4-bit and needs to drop to roughly 3 bits per weight to squeeze in, which is where quality degradation stops being theoretical.
• 64 GB module — 70B at Q4_K_M fits at 45.5 GB with about 12 GB spare for the KV cache and the runtime. This is the headline capability of the part and it is real, subject to the speed section below.
• Neither runs a 123B-class model at 4-bit; 80 GB of weights exceeds the larger module before any context at all.

Add the KV cache on top. The per-token cost is 2 x layers x kv_heads x head_dim x bytes_per_element; for a 70B-class model with 80 layers, 8 key-value heads and head dimension 128 at float16, that is 2 x 80 x 8 x 128 x 2 = 327,680 bytes per token, or 320 KiB. An 8,192-token conversation costs 2.6 GB and a 32,768-token one costs 10.5 GB — which is most of the 12 GB headroom, and the reason long-context work on this module usually needs a quantized KV cache.

The speed ceiling, derived

Decode reads every weight once per token, so the upper bound is bandwidth over weight bytes. With NVIDIA’s published 204.8 GB/s:

204.8 / 8.5  = ~24 tokens/second   (13B at Q4_K_M)
204.8 / 22.1 = ~9  tokens/second   (34B at Q4_K_M)
204.8 / 45.5 = ~4.5 tokens/second  (70B at Q4_K_M)

These are ceilings derived from a datasheet peak, not measurements. Real rates are lower — no memory subsystem achieves its theoretical peak, and the runtime adds attention over the growing KV cache on top of the weight read. What the numbers establish is the shape of the decision: the 70B fits, and at best it produces about four tokens a second, which is fine for a batch summarisation job running overnight and unusable for anything a person is waiting on.

Prompt processing behaves differently and this is where the 200 or 275 TOPS shows up. Prefill runs the whole prompt through in parallel and is compute-bound, so a long prompt is comparatively cheap here while the answer is comparatively expensive — the same asymmetry that shapes reported tokens-per-second figures everywhere else, just with an unusually large gap between the two rates.

The runtime choice interacts with this more than it does on a CPU-only board. llama.cpp built with the CUDA backend is the low-friction route and gets you the bandwidth-bound number above. NVIDIA’s own TensorRT-LLM path compiles an engine ahead of time for the specific model, precision and batch shape, which mostly buys prefill throughput and batching efficiency rather than single-stream decode speed — because single-stream decode is bounded by the same 204.8 GB/s whatever compiles it. If your workload is one user waiting for an answer, the engine build is unlikely to move the number that annoys you. If it is many concurrent requests, it moves a different number a great deal: batching amortises the weight read across requests, which is the one documented way out of the bound.

Power mode changes the answer

The datasheet gives the 32 GB module a 15–40 W range and the 64 GB module 15–60 W, and the module boots into a mode, not into its maximum. On JetPack, nvpmodel selects it and jetson_clocks pins clocks to the ceiling of the selected mode:

sudo nvpmodel -q          # which mode am I in
sudo nvpmodel -m 0        # mode 0 is the maximum-performance mode
sudo jetson_clocks        # pin clocks within that mode
sudo tegrastats           # live power, clocks and memory use

A benchmark taken in a 15 W mode and one taken in the 60 W mode are different measurements of different machines, and the memory controller clock is among the things that moves. Before comparing any figure — yours or somebody else’s — check nvpmodel -q. A reported tokens-per-second figure with no power mode attached is missing the single largest variable on this hardware.

tegrastats is also how you check the memory budget in the section above against reality rather than against arithmetic. It reports RAM use as a fraction of the total pool, and because that pool is shared there is no separate GPU memory figure to read — the model, the OS and every camera buffer appear in the same number. Watch it during a load: the peak while weights are being read and laid out is higher than the steady state, and a configuration that fits at rest can fail at load time on a module that looked like it had headroom.