Building an Offline AI Workstation

Parts lists go out of date and depend on where you are and what you can get. A method does not. Specify the workload, derive the memory requirement, and let that requirement do the shopping — which also gives you a defensible answer to “why this one”.

Specify backwards

The common failure is to buy hardware and then discover what it runs. Reverse it. Write down, in order:

• The tasks. Coding assistance, document processing, local chat, fine-tuning, image work. Each has a different profile, and one of them — training — has a memory profile an order of magnitude above the rest.
• The size class those tasks need, which the small-models and quality pages give you a way to decide. Be honest: most people specify for the largest model they might one day want and use a mid-size one daily.
• Your real context length. Measure it from work you already do. This term is frequently as large as the choice of quantisation.
• Concurrency. One user is one sequence. Anything shared changes the arithmetic entirely and belongs in the team serving model instead.
• Whether you will train. Even a QLoRA fine-tune has a different memory profile from inference, and it is the one requirement that will double your specification.

Deriving the memory requirement

Same arithmetic as everywhere in this cluster, run in the direction of a purchase decision:

required_vram = weights + kv + overhead

weights = params * bits_per_weight / 8
kv      = 2 * layers * kv_heads * head_dim * ctx * elem_bytes
overhead ~ 1-2 GiB

# example: a 32B model at Q4_K_M, 16k context,
#          64 layers, 8 kv heads, head_dim 128, fp16 cache
weights = 32e9 * 4.8 / 8            = 19.2 GB  -> 17.9 GiB
kv_tok  = 2 * 64 * 8 * 128 * 2      = 256 KiB
kv      = 256 KiB * 16384           =  4.0 GiB
overhead                            =  1.5 GiB
                                      --------
required                              23.4 GiB

# 24 GB is uncomfortably tight; an 8-bit KV cache buys 2 GiB back,
# or drop to 8k context, or accept Q4_K_S.

Run that for the largest model you actually intend to use daily, then again for the one you would like to use occasionally. The gap between those two answers is the decision, and seeing it as a number usually settles an argument that otherwise runs for a week.

If fine-tuning is on the list, add a second calculation: a QLoRA run needs the four-bit base plus adapters plus activations, and activations scale with sequence length and batch size. In practice this means budgeting well above the inference figure for the same model.

Which constraints actually bind

In rough order of how often each one is the thing that ruins a build:

• VRAM capacity. Binary. Either the model fits or you pay a large, non-linear penalty for spilling to system memory. This outranks everything else and is the number to specify against.
• Memory bandwidth. Sets generation speed, because each token streams the weights. Two cards with the same capacity and different bandwidth are not the same machine, and bandwidth is the spec people forget to look up.
• System RAM. Wants to be comfortably larger than VRAM. Models are loaded through it, and mixed CPU/GPU execution needs somewhere to put the overflow.
• Storage. Fast NVMe, and more of it than you expect — checkpoints are tens of gigabytes each and a habit of trying models fills a drive quickly. Load time is dominated by disk throughput.
• Power and thermals. Sustained inference is a sustained load, unlike gaming. Budget headroom in the supply, plan the airflow, and be realistic about noise if this sits where you work.
• Compute throughput. Last, genuinely. It affects prompt processing and training more than it affects generation, and it is rarely what stops you.

One large card or several small ones

The tempting arithmetic is that two cards of capacity N equal one card of 2N. They do not, and the differences run in both directions.

• Splitting a model across cards costs interconnect traffic at every layer boundary. Over a fast direct link this is modest; over ordinary PCIe lanes it is noticeable, and the effect grows with the number of cards.
• Not all software handles it equally. Batching servers implement tensor parallelism properly and generally want a power-of-two card count; local runtimes split by layers, which works but scales less well.
• Two cards are excellent for two models. A generation model on one and an embedding or draft model on the other, with no contention, is often more useful in practice than one larger model.
• Unified-memory machines are a third option with different arithmetic entirely: very large capacity, moderate bandwidth. Large models load and generate slowly. Excellent for running something that would otherwise be impossible; poor for throughput.

The software side

The part that determines whether the machine is pleasant to own.

• Pin the stack. Driver, CUDA or equivalent runtime, and inference engine versions in a written record. Upgrades break things, and reconstructing a working combination from memory is miserable.
• Containers for the serving stack, so an experiment cannot break the working setup.
• One model directory, symlinked into whichever tools you use, rather than three copies of every checkpoint.
• Genuinely offline means genuinely offline. If this is an air-gapped build, mirror the model files, the tokenizers and the package indexes deliberately. A great deal of tooling reaches for the network on first run and fails in confusing ways when it cannot.
• Record the hash of every model file you install, for the same reasons as everywhere else in this cluster.

Before you buy anything

Rent the equivalent for a few hours first. Cloud GPU instances exist at every capacity, and an afternoon spent running your real workload on the specification you are considering will tell you more than any amount of reading — including, quite often, that a smaller card is sufficient, or that the model you were specifying for is not the one you actually enjoy using.

Then be clear with yourself about why you are buying. If the reason is privacy, air-gapping, or an existing machine that is idle, the arithmetic above is the whole job. If the reason is cost, run the cost model first — for a single user the utilisation is usually low enough that the machine is being justified by something else, and it is worth knowing which something it is before the parts arrive.