Molecule Generation Models Explained

A generative model for molecules samples from a learned distribution over chemical structures. The interesting engineering is not in the sampling — it is in the fact that most strings are not molecules, most molecules are not useful, and most useful molecules cannot be made.

What the model is actually emitting

Three output formats dominate, and each fails differently.

• SMILES strings. A language model over chemical text. Marwin Segler and colleagues showed a character-level recurrent network trained on ChEMBL could generate valid, drug-like structures in “Generating Focused Molecule Libraries for Drug Discovery with Recurrent Neural Networks” (ACS Central Science, 2018). Simple, and it can emit strings that do not parse.
• Graphs, built incrementally. Add an atom, add a bond, repeat. Valence can be enforced at each step by masking illegal actions, so invalid output is impossible — at the cost of a fiddlier decoder and a generation order the model has to learn.
• Fragments or junction trees. Generate a tree of chemically sensible substructures and then assemble them. Wengong Jin, Regina Barzilay and Tommi Jaakkola’s junction tree variational autoencoder builds molecules from a vocabulary of rings and linkers extracted from the training set, which makes every output valid and biases it toward familiar chemistry.

The latent-variable framing that made the field visible came from Rafael Gómez-Bombarelli and colleagues in “Automatic Chemical Design Using a Data-Driven Continuous Representation of Molecules” (ACS Central Science, 2018): encode SMILES into a continuous space, optimise a property in that space with a surrogate model, decode. Its headline weakness — that points near a valid molecule often decode to garbage — motivated most of what followed.

Worked: what a validity check tests

“Validity” in every published benchmark means one thing: the string parses and sanitises in RDKit.

from rdkit import Chem

candidates = [
    "CC(=O)Oc1ccccc1C(=O)O",   # aspirin
    "c1ccccc1c",               # dangling aromatic atom, ring not closed
    "CC(C)(C)(C)C",            # five bonds on a neutral carbon
    "C1CCCCC",                 # ring-opening 1 never closed
]

for smi in candidates:
    mol = Chem.MolFromSmiles(smi)   # returns None on failure
    print(smi, "->", "valid" if mol else "invalid")

# valid, invalid, invalid, invalid

Each failure is a different rule. The second string opens an aromatic system that never closes. The third violates carbon’s valence, which the parser accepts syntactically and sanitisation rejects. The fourth leaves a ring-closure digit unmatched. A character-level model trained on SMILES has to learn all three constraints — balanced parentheses, matched ring digits, valence arithmetic — from examples, with no structural help.

The check that matters in a pipeline is slightly stricter than MolFromSmiles returning non-None: strip salts to the largest fragment, neutralise where appropriate, canonicalise, and then deduplicate on the canonical string or the InChIKey. A generator reporting 100% validity that emits the same twelve molecules repeatedly has told you nothing.

Making validity unconditional

SELFIES — self-referencing embedded strings, introduced by Mario Krenn and colleagues in Machine Learning: Science and Technology (2020) — is a line notation whose grammar carries a derivation state. Each token is interpreted relative to the remaining valence of the atom being extended, and a token that would overflow is clamped rather than rejected. The result is that every SELFIES string decodes to a valid molecule, including random ones, so a model emitting SELFIES has a validity rate of 100% without learning anything.

That is genuinely useful for latent-space optimisation, where you need arbitrary points to decode. It also demonstrates why validity is the weakest metric in the field: it is now free, and a metric you can saturate by changing the output alphabet was never measuring the thing you cared about.

Goal-directed generation and reward hacking

Unconditional generation is a warm-up. The real task is conditional: produce molecules that score well on some objective — a predicted activity, a docking score, a multi-property profile — usually via reinforcement learning or iterated fine-tuning on high-scoring samples.

The failure is structural, not incidental. The objective is a model, and optimising hard against a model finds its blind spots. Push a generator against a QSAR classifier and it will produce molecules that score brilliantly and sit far outside the classifier’s applicability domain, where the score means nothing. Push it against a docking score and it will grow large greasy molecules that fill the pocket and maximise the sum of contact terms, because the scoring function rewards contact and the physical reasons that molecule would never work are not in it.

The mitigations are all forms of constraint: bound the objective with a similarity term to a reference series, add explicit property filters, use an ensemble as the oracle and penalise disagreement, cap molecular weight and lipophilicity. None of them changes the fact that the score is a proxy, so the output is a set of candidates for a laboratory to test.

The metrics, and what each one misses

Two benchmark suites are standard — GuacaMol from Nathan Brown and colleagues, and MOSES from Daniil Polykovskiy and colleagues — and both report a similar panel.

• Validity. Parses and sanitises. Free with SELFIES or a graph decoder; near-free with a well-trained SMILES model.
• Uniqueness. Distinct canonical structures among k samples. Catches mode collapse, which is the most common actual failure.
• Novelty. Fraction not in the training set. Trivially maximised by generating nonsense, so it is only meaningful next to a quality measure.
• Fréchet ChemNet Distance. Compares the distribution of generated molecules to a reference in the activation space of a pretrained network, in the spirit of the FID used for images. Sensitive to distribution shift that per-molecule metrics miss.
• Internal diversity. Mean pairwise Tanimoto distance within the generated set. Low diversity plus high novelty means the model found one strange corner of chemical space and stayed there.

The constraint that decides usefulness

A generated molecule you cannot make is a picture. Wenhao Gao and Connor Coley examined this directly in “The Synthesizability of Molecules Proposed by Generative Models” (Journal of Chemical Information and Modeling, 2020), running retrosynthesis software over generative-model output and finding that a substantial share of proposals had no route.

Two families of fix are in common use. Heuristic scores — the synthetic accessibility score of Peter Ertl and Ansgar Schuffenhauer, based on fragment frequency and structural complexity — are cheap enough to include in a reward function and crude. Search-based filtering runs an actual retrosynthesis planner over candidates and keeps only those with a route to purchasable starting materials, which is expensive and much more honest. The strongest form removes the problem: generate only molecules assembled by reactions from a defined set applied to catalogued building blocks, so the route exists before the molecule does.