Hugging Face transformers: Load and Run a Model

Running a model with transformers is three objects — a tokenizer, a model, a generation call — and two steps that most tutorials skip and that account for most of the confusing results: applying the chat template, and choosing the precision deliberately rather than accepting the default.

Three objects and one loop

The from_pretrained pattern has been the library’s interface for years and is as stable as anything in this ecosystem gets. The auto classes inspect the repository’s configuration and instantiate the right implementation, so the same four lines work across architectures.

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model     = AutoModelForCausalLM.from_pretrained(MODEL_ID, device_map="auto")

inputs  = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

One detail in that snippet causes a steady stream of confusion: generate returns the prompt tokens followed by the new ones, so decoding the whole tensor prints your prompt back at you. Slice off the input length, or use the tokenizer’s batch decode on the generated portion only.

prompt_len = inputs["input_ids"].shape[-1]
new_tokens = outputs[0][prompt_len:]
print(tokenizer.decode(new_tokens, skip_special_tokens=True))

The chat template is not optional

This is the highest-value paragraph on the page. An instruction-tuned model was trained with a specific format of special tokens marking system, user and assistant turns. Feeding it a bare string skips that format entirely, and the model responds — badly, in a way that looks like the model being poor rather than like a formatting bug.

messages = [
    {"role": "system", "content": "You are terse."},
    {"role": "user",   "content": "Explain KV caching in two sentences."},
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,   # opens the assistant turn
)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

The template ships with the model, so switching models switches formats automatically — which is exactly why hand-writing the format strings is a mistake. Two symptoms tell you the template is missing: the model continues your prompt rather than answering it, and it rambles past where an answer should end because the stop token it was trained to emit never appears in a format it recognises. If a repository has no template, the model is probably a base model rather than an instruction-tuned one, which is a different tool — see base versus instruct models.

Will it fit? The arithmetic

Work this out before downloading forty gigabytes. Every term is countable and none of it depends on a benchmark.

weights = parameters × bytes_per_parameter

  float32  4 bytes    bfloat16  2 bytes    8-bit  1 byte    4-bit  0.5 bytes

  a 7B model:   7e9 × 2 = 14.0 GB at bfloat16
                7e9 × 4 = 28.0 GB at float32      <- the trap
                7e9 × 1 =  7.0 GB at 8-bit

KV cache per token, per sequence:

  bytes = 2 × layers × kv_heads × head_dim × bytes_per_element
          ^ one each for K and V

  worked example: 32 layers, 8 kv heads, head_dim 128, bfloat16
  = 2 × 32 × 8 × 128 × 2 = 131,072 bytes ≈ 128 KB per token

  4,000 tokens of context  ≈  0.5 GB
  32,000 tokens            ≈  4.0 GB

Total ≈ weights + KV cache + activations + framework overhead
      (allow 1–2 GB for the last two on a single-stream run)

Read the layer count, head counts and head dimension from the model’s config file rather than guessing; they are all in it. Note that models using grouped-query attention have far fewer key-value heads than attention heads, which is the difference between a KV cache that fits and one that does not — the mechanism is covered in the KV cache.

Precision and device placement

Two arguments decide whether the model loads and how fast it runs. Both have caused enough confusion to be worth stating carefully.

Precision. Loading in the model’s native reduced precision rather than upcasting to float32 halves the memory and is almost always what you want on modern hardware. The keyword argument that controls this was renamed in recent versions of the library — older code passes torch_dtype, newer code passes dtype — so check which one your installed version expects rather than assuming. The value "auto" takes the precision recorded in the model’s own config, which is usually the right answer.

Placement. device_map="auto" spreads the model across available devices and, if it does not fit, will place layers on CPU or disk. That is a feature and a trap: the model loads, and generation is thirty times slower than expected because half of it is being read from CPU memory on every token. If throughput collapses, check where the layers actually went before blaming anything else.

Generation settings that change the output

The generation arguments have been stable for years and a handful of them account for nearly all the behaviour.

Batching, padding and the side that breaks it

Batching several prompts into one call is the easiest throughput win available, and there is one rule that decides whether it works: for decoder-only generation, padding must be on the left.

The reason is mechanical. Generation continues from the last position of the sequence. With right padding, the last positions are pad tokens, so the model continues from padding and produces nonsense for every sequence shorter than the longest. Set the tokenizer’s padding side to left before batching, and set a padding token if the tokenizer has none — many causal models ship without one, and reusing the end-of-sequence token for the purpose is the conventional fix.

The symptom is distinctive and worth memorising: the longest prompt in the batch produces a good answer and the shorter ones produce garbage. That is left-padding, every time.

What it is genuinely best at

Text generation is the thing transformers is worst at relative to the alternatives, and it is the only thing most tutorials show. Two other uses have no comparable alternative, and both are reasons to keep it installed even after you serve models elsewhere.

Raw logits. Calling the model directly rather than through the generation helper returns scores over the vocabulary for every position. That gives you a probability for a specific continuation — which is how you build a classifier that outputs a calibrated score rather than a word, and how you compute perplexity over a corpus. No hosted API exposes this fully, and where one exposes logprobs it is a truncated view. The mechanics of what those numbers mean are in logprobs explained.

out = model(**inputs)          # no generate(); one forward pass
out.logits.shape               # (batch, sequence, vocab)

# Score of a specific next token, as a probability:
probs = out.logits[0, -1].softmax(dim=-1)
probs[tokenizer.encode(" yes", add_special_tokens=False)[0]]

Hidden states. Requesting them returns the internal representations at every layer, which is the starting point for probing, feature extraction and any analysis of what a model is representing. For embeddings specifically, use a model trained for it rather than pooling a generative model’s hidden states — the reasons are on the Sentence Transformers page — but the mechanism is the same.

The third is fine-tuning, which lives in the surrounding ecosystem rather than in this library alone but starts from the same loaded model object. Whether it is the right move at all is a separate question, addressed in should you fine-tune.

What this library is not for

It is a research and experimentation library. For serving — concurrent users, continuous batching, paged attention, a stable HTTP endpoint — a dedicated inference server is not an optimisation, it is a different category of software. A naive loop over generate behind a web framework will serve requests one at a time and idle the GPU between tokens.

Use transformers to test whether a model can do the task, to compute embeddings or logits, and to fine-tune. Then serve it with vLLM or TGI, both of which load the same checkpoints and speak HTTP. The throughput difference between those and a naive loop is not marginal; it is the reason those projects exist, and the underlying mechanism is continuous batching.