Jetson Orin Nano Memory Budget for Local Models, Derived

On a desktop card the operating system lives somewhere else. On a Jetson module it lives in the same pool as the model, and that single difference is what most capacity estimates for this board get wrong.

What NVIDIA publishes

NVIDIA’s developer blog post announcing the Super mode for the Jetson Orin Nano Developer Kit, published 17 December 2024, gives the module as 8 GB of 128-bit LPDDR5 with 68 GB/s of memory bandwidth, rising to 102 GB/s under the Super power mode introduced in JetPack 6.2, alongside a move from 40 to 67 sparse INT8 TOPS, an added 25 W power mode on top of the existing 7 W and 15 W, and a price reduction to $249.

Three of those figures matter for running a language model, and it is worth being clear which. The 8 GB decides what fits. The 68 or 102 GB/s decides the token ceiling. The TOPS figure decides almost nothing for this workload, because single-stream generation is memory-bound and leaves most of the tensor throughput idle — TOPS is the number that sells edge boards and the one least connected to tokens per second.

Shared memory changes the budget

The 8 GB is one physical pool addressed by both the CPU and the GPU. There is no separate VRAM to allocate from, which has one advantage and one large disadvantage.

The advantage is that there is no host-to-device copy: a tensor written by the CPU is visible to the GPU without a transfer, so the load path is shorter than on a discrete card and the PCIe arithmetic on the lanes page does not apply at all.

The disadvantage is that the operating system, the desktop if one is running, the page cache and every other process draw on the same 8 GB the model wants. A stock Linux install with a graphical session is typically using 1.5–2.5 GB before anything else starts. Running headless with the graphical target disabled recovers most of that, and on an 8 GB module the difference between a desktop session and a headless one is the difference between a 3B model fitting and not.

8 GB module  = 7.45 GiB physical

  minus firmware/carveout reserved regions   ~0.3 GiB
  minus kernel + base userspace              ~0.8 GiB
  minus graphical session (if running)       ~1.0 GiB
  ------------------------------------------------
  usable, headless      ~6.3 GiB
  usable, with desktop  ~5.3 GiB

Those subtractions are estimates of a stock configuration and will differ on yours; the point is the structure of the sum, not the exact figures. Read your own with free -m before the model loads, which is the last section below.

What fits, derived

Weights from the published GGUF file sizes, cache from the models’ own layer and head counts, against 6.3 GiB headless:

Llama 3.2 1B Q4_K_M   0.81 GB -> 0.75 GiB weights
                       32 KiB/token cache -> 5.4 GiB free = ~177k tokens
                       (model caps at 131,072)

Llama 3.2 3B Q4_K_M   2.02 GB -> 1.88 GiB weights
                      112 KiB/token cache -> 4.2 GiB free = ~38k tokens

Llama 3.2 3B Q8_0     3.42 GB -> 3.19 GiB weights
                      112 KiB/token cache -> 2.9 GiB free = ~26k tokens

Llama 3.1 8B Q4_K_M   4.92 GB -> 4.58 GiB weights
                      128 KiB/token cache -> 1.5 GiB free = ~12k tokens
                      (leaves nothing for compute buffers - marginal)

File sizes are the ones published on bartowski’s GGUF model cards (retrieved 11 August 2026); the per-token cache figures come from the published configs — 16 layers with head dimension 64 for the 1B, 28 layers with head dimension 128 for the 3B, 32 layers with head dimension 128 for the 8B, all with 8 KV heads.

The honest reading: a 3B at Q4 is this module’s comfortable working size, a 3B at Q8 fits with a usable context, and an 8B at Q4 is the edge of the envelope — it will load headless at a short context and will fail the moment anything else on the board wants a gigabyte. Quantizing the cache to eight bits with --cache-type-k q8_0 --cache-type-v q8_0 roughly doubles the context available in each of those rows and is close to free on this device.

What happens when the budget is exceeded is worse on a shared pool than on a discrete card, and it is the reason to leave margin rather than to fill the last gigabyte. A discrete card returns CUDA error: out of memory, the allocation fails, the process exits, and the machine is otherwise fine. Here there is no separate pool to fail against: the allocation succeeds, the kernel reclaims pages from everything else, and the board goes to swap or the out-of-memory killer picks a victim — possibly the SSH daemon, possibly the inference process, possibly something you needed. On a headless module reached over the network, that is the difference between an error message and a trip to fetch a keyboard.

Swap deserves a specific warning on this class of device. A default install may have zram configured, which compresses pages in RAM rather than writing them out, and that is fine. Swap to the eMMC or an SD card is not: model weights compress poorly and are re-read on every token, so a model that has partly spilled to card-backed swap generates at a rate no arithmetic on this page predicts, while wearing the card. Size the model to fit in RAM and leave swap for everything else.

The bandwidth ceiling, and the Super mode

The same bound applies as everywhere else: every token reads every weight, so the rate cannot exceed bandwidth over weight bytes.

ceiling = bandwidth / weight_bytes

  at 68 GB/s   3B Q4_K_M (2.02 GB) -> 33.7 tok/s
               8B Q4_K_M (4.92 GB) -> 13.8 tok/s

  at 102 GB/s  3B Q4_K_M (2.02 GB) -> 50.5 tok/s
               8B Q4_K_M (4.92 GB) -> 20.7 tok/s

Derived ceilings, not measurements. The Super mode is a 50% increase in the bound, which is a genuine and unusually large software-only gain — but it comes with the 25 W power mode, and a module in a passively cooled enclosure that cannot dissipate 25 W will not hold those clocks. The power mode is selected with nvpmodel and the current clocks are visible through tegrastats; a mode that the thermal solution cannot sustain shows up there as clocks well below the mode’s nominal.

Measuring the budget on the module

1. Get the real starting point before anything loads: free -m for the pool, and systemctl isolate multi-user.target to drop the graphical session if you want the headless figure.
2. Confirm the power mode and the clocks it is actually holding: sudo nvpmodel -q to read the mode, then tegrastats during a sustained load to see whether the clocks hold or fall.
3. Load the model with an explicit context length rather than the model’s advertised maximum — -c 8192 — and watch free -m during the load. On shared memory an over-large cache allocation does not fail cleanly; it takes the system into swap or the out-of-memory killer.
4. Benchmark generation and prefill separately: llama-bench -m model.gguf -ngl 99 -p 512 -n 128 -r 5, and compare the generation figure against the ceiling above.
5. Re-run with a quantized cache and see how much context that buys at the same rate. On an 8 GB shared pool this is usually the single highest-value change available.