Comparing Representations Across Models

Two models trained on the same data with different seeds: do they represent the world the same way? The question sounds empirical and turns out to be mostly a question about which transformations you decide not to care about. Every similarity measure is a different answer to that, and the answer determines what a high score means.

The question, and why it is awkward

You have two activation matrices, each with one row per input and one column per unit. Same rows, different columns — possibly a different number of them. You want a single number saying how similar the two representations are.

The naive comparisons all fail immediately. Correlating unit i with unit i assumes the two networks put the same thing in the same place, which they do not: units are arbitrarily ordered, and if superposition is right they are not the right unit of comparison in the first place. Euclidean distance between the matrices is undefined when the widths differ, and meaningless when they do not, because the two spaces have no shared basis.

So every usable measure begins by declaring a set of transformations that should not change the answer. Permute the units: still the same representation. Rescale everything: probably still the same. Apply an arbitrary invertible linear map: now it depends entirely on what you are asking.

Invariance is the whole design decision

Here is the trade-off, stated as sharply as it can be. A measure invariant to all invertible linear transformations will call two representations identical whenever a linear map takes one to the other. That is often what you want — but note that a linear probe can also read anything through such a map, so a measure with that invariance cannot distinguish representations that differ in how easily a downstream layer can use them.

A measure invariant only to orthogonal transformations and isotropic scaling keeps that distinction: it cares about the geometry, not just the span. It will report that two representations differ when one stretches a direction the other compresses, which matters because downstream layers are sensitive to exactly that.

The measures

Linear CKA, in full

Short enough to write out, which is the point — there is nothing hidden in it.

import torch

def linear_cka(X, Y):
    """X: (n, p), Y: (n, q). Same n inputs, any widths. Returns [0, 1]."""
    X = X - X.mean(0, keepdim=True)          # centring is not optional
    Y = Y - Y.mean(0, keepdim=True)

    xty = (X.T @ Y).norm(p="fro") ** 2
    xx  = (X.T @ X).norm(p="fro")
    yy  = (Y.T @ Y).norm(p="fro")
    return (xty / (xx * yy)).item()

# sanity checks you should run before trusting any number it gives you
n, p = 512, 64
X = torch.randn(n, p)

print(linear_cka(X, X))                              # 1.0    identical
print(linear_cka(X, X[:, torch.randperm(p)]))        # 1.0    permuted units
print(linear_cka(X, 7.3 * X))                        # 1.0    rescaled
print(linear_cka(X, X @ torch.linalg.qr(
        torch.randn(p, p))[0]))                      # 1.0    rotated
print(linear_cka(X, torch.randn(n, p)))              # ~0.1   unrelated
print(linear_cka(X, X @ torch.randn(p, p)))          # < 1.0  general linear map

That last line is the informative one. CKA is not invariant to a general invertible linear map, and that is a deliberate design choice rather than a shortcoming. It is also why CKA and CCA can disagree sharply on the same pair of representations while both being implemented correctly.

The centring step deserves its comment. Uncentred, the measure is dominated by the mean activation, and two representations with similar means but different structure will score close to one. Papers have been confused by this.

Where the measures disagree

CKA became the default after the 2019 work showed it recovers the expected structure — corresponding layers of independently trained networks score highest — where CCA-based measures often did not. Subsequent work has been more critical, and the criticisms are worth knowing before you rest a claim on a score.

• Dominant directions drive the score. Because CKA is built from Frobenius norms of Gram matrices, a few high-variance directions can determine the result. Two representations can score high while differing substantially in everything outside the top components — and those low-variance directions may be precisely where a rare but important feature lives.
• It is sensitive to the input distribution. Change which inputs you feed and the score changes, sometimes a lot. A similarity number is a statement about a representation on a dataset, and comparing scores computed on different datasets is not valid.
• Measures rank pairs differently. Given three models, CKA and a CCA variant will not necessarily agree on which two are most alike. If your conclusion flips when you change measure, you do not have a conclusion.

The defensible protocol is boring: report at least two measures with different invariances, on a stated input distribution, with a random baseline for scale.

What “models converge” claims rest on

There is a recurring and interesting claim that as models get larger and are trained on more data, their internal representations become more similar to each other — across seeds, across architectures, and even across modalities, so that a vision model and a language model come to encode a shared structure. Huh and colleagues argued a strong version of this in 2024.

Take it seriously and read the method carefully, because everything depends on the measurement. A convergence claim is a claim about a similarity score rising, so it inherits every property of the measure used. If the measure is dominated by high-variance directions, then “representations converge” may mean “the dominant directions converge”, which is a weaker and much less surprising claim — large models trained on overlapping data agreeing about the coarse structure of the world is close to what you would expect.

The questions to ask of any such result: which measure, on which inputs, against which baseline, and does the claim survive a measure with different invariances? A convergence result that holds under CKA, a CCA variant and nearest-neighbour overlap simultaneously is strong. One that holds under a single measure is a finding about that measure until shown otherwise.

Where this is genuinely useful

• Detecting what fine-tuning changed. Compare a fine-tuned model to its base layer by layer. A sharp drop in similarity at particular layers tells you where the adaptation happened, which is a concrete diagnostic when a fine-tune has regressed something.
• Checking a distillation. If a student is supposed to be reproducing a teacher’s computation and not merely its outputs, layer-wise similarity is the direct check, and it is informative in a way output agreement is not.
• Locating redundancy. Adjacent layers with very high similarity are candidates for pruning or merging, which turns this into a practical tool rather than an analysis.
• Verifying a training change. A modification that was meant to change how the model represents something, and does not move layer-wise similarity at all, probably did not do what it was supposed to.