Predicting Gene Expression From DNA Sequence

A sequence-to-expression model takes DNA and nothing else and predicts how active that DNA will be. Everything interesting about the architecture follows from one number: how far away the model is allowed to look.

The task, stated precisely

The input is a window of reference genome, one-hot encoded as four channels — A, C, G, T — by however many bases wide the window is. The output is not a single expression value but a set of tracks: for each of thousands of experimental assays across many cell types, a predicted signal along the window, binned at some resolution. Those assays include transcription measured by CAGE, chromatin accessibility by DNase or ATAC, and histone modifications by ChIP.

Two things follow from that framing and both matter. The model is multi-task by construction, which is why it works at all — the assays share underlying regulatory grammar, so a model forced to predict all of them learns features that a single-track model would not have the data to find. And the output is spatial, a profile rather than a number, which means the loss rewards putting signal in the right place, not just getting the total right.

The training set is the reference genome partitioned into windows, with chromosomes held out for validation and test. That partition is the only thing standing between this and total leakage, and it is why held-out chromosome performance is the number these papers report.

What is in a promoter window

The region immediately around a transcription start site is where the strongest and most legible signal lives. Its contents:

• Core promoter elements. A TATA box, when present, sits roughly 25 to 30 bases upstream of the start site — and it is present in a minority of human promoters, which is itself the useful fact. An initiator element overlaps the start site. Downstream promoter elements sit inside the transcript.
• CpG islands. Most human promoters sit in regions of unusually high CG dinucleotide density. This is a compositional feature a model picks up almost immediately, and it is why a convolutional first layer trained on this task learns GC-rich filters before it learns anything specific.
• Transcription factor binding motifs. Short, degenerate patterns, typically 6 to 12 bases, that specific proteins recognise. Their arrangement — which motifs, how many, how far apart, in which orientation — is the regulatory grammar the model is really learning.
• Nothing about the cell. This is the constraint people forget. The input is the reference sequence, which is identical in every cell type. Cell-type specificity has to come out of the output head, which predicts a different track per cell type from the same input. The model is learning which sequences are active where, not reading a cell state.

A motif scan, worked

The first convolutional layer of these models is doing something very close to a classical position-weight-matrix scan, and working one by hand makes the layer legible.

A position-weight matrix stores, for each position in the motif, the log-odds of each base: the log of the probability of that base at that position in known binding sites, divided by its background frequency. Take a six-position matrix and score the sequence TGACGT:

PWM (log2 odds)     pos1   pos2   pos3   pos4   pos5   pos6
  A                 -2.0   -1.8    1.9   -2.1   -1.7   -2.0
  C                 -1.9   -2.0   -2.2    1.8   -1.9   -1.8
  G                 -1.8    1.9   -2.0   -2.0    1.7   -1.9
  T                  1.9   -2.1   -1.9   -1.8   -2.0    1.9

score("TGACGT") = 1.9 + 1.9 + 1.9 + 1.8 + 1.7 + 1.9 = 11.1
score("TGTCGT") = 1.9 + 1.9 - 1.9 + 1.8 + 1.7 + 1.9 =  7.3
score("AAAAAA") = -2.0 -1.8 +1.9 -2.1 -1.7 -2.0     = -7.7

Slide that across the window and you get a score per position; take the maximum, or a soft sum, and you have a feature meaning “this motif is present, and this strongly”. A convolutional filter over the one-hot encoding computes exactly this dot product, with the weights learned rather than counted from known sites. The single-mismatch case above is the important one: the motif is degenerate, one substitution costs about 3.8 bits and the site still scores well above background, which is why binding is a graded property and why a model that treats motifs as present-or-absent underperforms one that keeps the score.

Receptive field is the architecture

Promoters are not the whole story. Enhancers regulate genes from tens or hundreds of kilobases away, looping to the promoter in three dimensions, and a model that cannot see them cannot explain most of the variation in expression. So the design question is how to build a receptive field large enough to include distal regulatory elements while keeping the resolution fine enough to localise them.

The two generations of answer are worth contrasting. Dilated convolutional stacks grow the receptive field exponentially with depth by skipping inputs at increasing stride, and reached the tens of kilobases. Enformer, published by Žiga Avsec and colleagues in Nature Methods in 2021, replaced the upper convolutional tower with a transformer trunk, taking a 196,608-base input window and predicting 896 output bins of 128 bases each. The reported effect of that change was a receptive field about five times larger, reaching sequence elements up to 100 kilobases from the start site, and correspondingly better variant-effect predictions.

The arithmetic of the output shape is worth reading off directly: 896 bins times 128 bases is 114,688 bases of predicted signal, centred inside a 196,608-base input. The difference is context the model is allowed to see but is not asked to predict, because a prediction at the edge of the window would have no context on one side. That padding is a general feature of this architecture family, not an Enformer quirk.

The gap between two kinds of accuracy

These models report strong correlations on held-out chromosomes, and that number answers a specific question: given two different loci in the genome, can the model tell which is more highly expressed? It can, well.

A different question is: given the same locus in two different people whose sequences differ by a handful of variants, can the model tell which person expresses it more highly? Published evaluations of sequence-to-expression models on personal genomes report that performance on this second question is substantially weaker than the headline across-locus numbers suggest, and in some evaluations close to uninformative. This is not a contradiction: across loci the variation is dominated by large, categorical differences in promoter and enhancer content, while across individuals it is dominated by small-effect variants in a fixed regulatory context, and a model can be excellent at the first while being blind to the second.

The practical reading. If you want to know which regions of a genome are regulatory, these models are a strong tool. If you want to know the effect of a specific variant on a specific person’s expression, check what the model was actually evaluated on before relying on it, and prefer methods evaluated against measured expression across individuals. No output from these models is a clinical interpretation of a variant, and treating a predicted effect size as evidence about an individual is exactly the extrapolation the personal-genome evaluations caution against.

Two further limits are structural. The models are trained on a reference genome and predict assays measured in bulk, so they inherit the cell types those assays were run in and nothing else — connecting their output to a cell population measured directly means going through something like single-cell expression analysis. And they see only the sequence in the window, so anything driven by distant elements outside it, by copy number, or by the epigenetic state a cell inherited rather than encoded, is out of scope by construction.