Steering Vectors and Activation Engineering

A steering vector is a direction in a model’s residual stream that, when added during a forward pass, shifts behaviour in a consistent way — more cautious, more formal, less inclined to refuse. It is one of the few interpretability results that turns directly into a control knob, and it comes with collateral damage that most write-ups do not measure.

What a steering vector is

The premise is the linear representation hypothesis: high-level properties are encoded as directions in activation space, so moving along one should change the property. Concretely, at layer L you replace the residual stream h with h + α·v, where v is a fixed vector with the same dimension as the model’s hidden size and α is a coefficient you choose. Nothing is retrained, nothing is stored, and the change costs one vector addition per token.

The idea is older than the current wave — word-embedding arithmetic in the word2vec era was the same claim about a much simpler representation. What is new is that it works on the internal states of a large model, at inference time, on behaviours you care about.

Building one from contrastive pairs

The reliable recipe is difference-of-means over a set of contrastive pairs, not a single pair.

1. Write matched prompt pairs. Each pair differs only in the property you want. Thirty to a few hundred pairs; more is better, and matching them tightly matters more than having many. If your positive prompts are systematically longer, you will extract a length direction.
2. Capture the residual stream at your chosen layer for every prompt, at a fixed position rule — usually the last token of the prompt.
3. Take the difference of the means. v = mean(positive) − mean(negative). Averaging is what cancels the incidental content of individual prompts and leaves the axis they differ on.
4. Normalise, and record the scale you normalised to. The useful range of α depends on the typical norm of the residual stream at that layer, which varies by layer and by model, so a coefficient that works in one place means nothing in another.

This construction is the same one used to build a probe direction, and that is not a coincidence — a probe that reads a property and a vector that writes it are the same object viewed from two sides. The difference is that steering tests the causal claim the probe cannot.

Where and when to inject it

The implementation

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

name = "gpt2"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name).eval()
layers = model.transformer.h                 # Llama-style: model.model.layers
LAYER = 6

@torch.no_grad()
def last_token_state(text, layer):
    ids = tok(text, return_tensors="pt")
    out = model(**ids, output_hidden_states=True)
    return out.hidden_states[layer + 1][0, -1]   # +1: [0] is the embedding

pairs = [
    ("Write a careful, hedged answer:", "Write a confident, blunt answer:"),
    ("Answer cautiously and note uncertainty:", "Answer decisively:"),
    # ...30+ tightly matched pairs
]

pos = torch.stack([last_token_state(a, LAYER) for a, _ in pairs])
neg = torch.stack([last_token_state(b, LAYER) for _, b in pairs])
v = (pos.mean(0) - neg.mean(0))
v = v / v.norm()

def steer(vec, alpha):
    def hook(module, args, output):
        hidden = output[0].clone()
        hidden[:, :, :] = hidden + alpha * vec.to(hidden.dtype)
        return (hidden,) + tuple(output[1:])
    return hook

prompt = tok("The safest thing to do with a new medication is", return_tensors="pt")

for alpha in (0.0, 2.0, 6.0, 15.0):
    h = layers[LAYER].register_forward_hook(steer(v, alpha))
    with torch.no_grad():
        out = model.generate(**prompt, max_new_tokens=40, do_sample=False)
    h.remove()
    print(f"--- alpha={alpha}")
    print(tok.decode(out[0], skip_special_tokens=True))

Sweeping α including zero is the whole point of writing it this way. The zero run is your control, and the sweep shows you where the effect starts and where the model falls apart. Both boundaries move when you change the layer.

The side effects, and how to measure them

This is the part that decides whether a steering result is worth anything. You have deliberately pushed the model off its own distribution. Something else has changed, and if you only evaluate the behaviour you were targeting you will not see it.

Measure three things, every time:

• The target behaviour, on held-out prompts that were not used to build the vector. Prompts from the construction set will over-report.
• General capability. Perplexity on a corpus unrelated to the target, or accuracy on a small multiple-choice benchmark, run at each α. Plot it on the same axis as the target. The honest result is a trade-off curve, not a point.
• Off-target behaviour. A vector built to make a model more cautious may also make it more verbose, more formal, and more likely to refuse things it should not. Write a small set of probes for the adjacent behaviours you did not intend to change.

The failure mode nobody reports: a steering vector built from prompts that share incidental structure steers on that structure. If every positive example is a question and every negative is an imperative, you have built a question direction and it will do something that looks like your target on the prompts you tested.

What has actually been shown

Published work in this area, described without inflation. Subramani and colleagues (2022) showed that a vector added to hidden states can drive a model to produce a specific target sentence. Turner and colleagues (2023) introduced activation addition using contrastive prompt pairs for steering style and topic in a decoder model. Li and colleagues (2023) applied a related intervention along truthfulness-associated directions found by probing attention head outputs. Rimsky and colleagues (2024) formalised the contrastive difference-of-means construction across a set of behavioural categories. Arditi and colleagues (2024) reported that refusal behaviour in several open chat models is mediated substantially by a single direction, such that removing that direction from the weights suppresses refusal.

That last result is the one to think carefully about, because it points both ways: it is strong evidence for the linear representation hypothesis at the level of a real behaviour, and it is a straightforward technique for stripping safety behaviour out of an open-weights model. If you are reasoning about what releasing weights commits you to, this is part of the answer.

When this beats a prompt, and when it does not

Steering is not usually the right tool, and the comparison is worth being blunt about. A system prompt is free, portable across providers, auditable by a human, and does not require the weights. Steering requires local weights, adds a hyperparameter with a narrow usable range, needs its own evaluation harness, and has to be re-derived for every model version.

It earns its place in three situations: when the behaviour is one the model resists in the prompt but the direction still exists internally; when you need the control to persist through a long generation where prompt influence decays; and when you are doing research and the point is to test whether a direction found by probing is causally used. For production behaviour control, a system prompt and an output guardrail do more, more cheaply, with an audit trail.