AI in Genomics: Sequence Models Applied Outside Language

Nothing here is medical advice, and nothing here should be used to interpret anyone’s genetic results. This page is about how these models work and what their outputs are and are not.

Why the architecture transferred

The setting that made language modelling work is a very large corpus of discrete sequences with structure at many scales, and a training task that needs no labels. Biology supplies that almost too neatly. Protein and nucleotide sequence databases are enormous, growing fast, and entirely unlabelled with respect to function. Mask out part of a sequence, train a model to predict it, and you have the same objective with none of the annotation cost that has always limited supervised work in this area.

What makes it more than an analogy is what the resulting representations contain. Protein language models trained purely to fill in masked residues develop internal structure that correlates with contacts, secondary structure and functional sites — properties nobody supplied as a label. The reason is the same one behind structure prediction from alignments: the training corpus is the output of evolution, and evolution has already done the labelling. Sequences that survive are ones where the constraints were satisfied, so the statistics of which residue appears where encode those constraints.

Hold on to that, because it is the whole basis for interpreting the outputs. These models have learned what is evolutionarily usual. Every downstream score is, at bottom, a statement about how unusual something is.

Where biology is not language

• The alphabet is tiny. Four nucleotides carry two bits each, against tens of thousands of word pieces. Each token carries far less information, so a genomic model needs vastly longer contexts to see the same amount of signal, and tokenisation choices — single bases, fixed k-mers, learned pieces — have consequences that have no clean analogue in text.
• The dependencies are much longer. A regulatory element can sit hundreds of thousands of bases from the gene it controls, with no marker in between saying so. This is the reason long-range architectures were adopted here early and aggressively.
• There are no word boundaries. Genes, exons and regulatory elements are not delimited. Where language gives a model whitespace for free, a genomic model has to infer segmentation as part of the problem.
• The strand symmetry is real. A sequence and its reverse complement are the same physical molecule read the other way. A model that does not respect that is learning the symmetry from data instead of getting it for free.
• Repetition is not noise. A large fraction of a mammalian genome is repeated sequence. In text, near-duplicate training data is a problem to remove; here it is the subject matter, and it wrecks naive train/test splits.

What these models are asked to do

What a variant score actually means

Take the most consequential case. A model outputs a number saying a particular missense variant is likely damaging. Unpacked, that number says: this position is highly conserved across the evolutionary record the model was trained on, and this substitution is one that is rarely or never observed there. That is a genuinely informative prior. It is not an observation about the person carrying the variant, it is not a measurement of protein function, and it is not a diagnosis.

Clinical variant interpretation frameworks reflect this directly. Computational predictions are admitted as one line of evidence with limited weight, to be combined with population frequency, segregation within families, functional assay results, and case-level data. A strong computational score does not on its own reclassify a variant, and the reason is not conservatism about new methods — it is that the score is answering a question about evolutionary constraint rather than about clinical consequence, and the two come apart. A variant can be at a conserved position and be tolerated in humans; a variant can be at a poorly conserved position and matter enormously in a specific context.

The same distinction, in its clinical form, is the subject of the evidence ladder for diagnostic models.

The evaluation trap

Two problems here are worth knowing about, because they are easy to miss when reading a strong reported result.

The first is circularity. Benchmarks for variant effect predictors are often built from curated clinical databases. Those databases were curated by people who, among other evidence, consulted earlier computational predictors. So a new predictor can be partly rewarded for agreeing with its predecessors rather than with reality. The cleanest escape is evaluation against deep mutational scanning — experiments that measure the functional effect of a large number of substitutions directly — because those labels are measurements and were not influenced by any predictor.

The second is that the impressive-looking evaluation is often not the useful one. A model can predict very well which genes are expressed highly and where, because that variation is large and largely driven by sequence features the model has learned. Predicting how the particular variants carried by one person change expression relative to another’s is a much smaller, much harder signal, and it is the one that would matter for interpreting an individual genome. A headline number from the first evaluation says almost nothing about performance on the second, and the two are reported in the same units.