Topic Modelling: LDA vs Embedding Clustering

Both methods hand you a list of clusters with words attached, and both look convincing in a screenshot. They rest on incompatible assumptions about what a document is, and the assumption — not the output quality — is what should decide which you use.

What LDA assumes

Latent Dirichlet Allocation (Blei, Ng and Jordan, JMLR 2003) is a generative story about how documents come to exist. Each topic is a probability distribution over the vocabulary. Each document is a probability distribution over topics. To produce a document you draw a topic for each word position, then draw a word from that topic. Fitting the model means inverting that story: given the documents, recover the distributions that most plausibly produced them.

Three properties follow directly. A document is a mixture — 60% finance, 30% regulation, 10% technology — which is the honest description of most real documents. Topics are distributions over words, so they are directly readable, and the Dirichlet priors give you explicit control over how concentrated documents and topics are. And the whole thing is bag-of-words: word order is discarded, and the model has no idea that car and automobile are related unless they co-occur with the same neighbours.

The practical weaknesses follow just as directly. LDA needs a reasonable amount of text per document — it is notoriously poor on tweets and short titles, because a ten-word document gives the sampler almost nothing to work with. It needs the number of topics chosen in advance. And it is sensitive to preprocessing in a way that feels unfair: leave function words in and every topic is dominated by them, which is one of the few places a stop list genuinely earns its place.

What embedding clustering assumes

The modern alternative — the arrangement BERTopic (Grootendorst, 2022) and Top2Vec (Angelov, 2020) popularised — makes a different bet: embed each document into a dense vector, reduce the dimensionality (usually UMAP), cluster the reduced vectors (usually HDBSCAN), then describe each cluster by the terms that are distinctive within it, scored with something TF-IDF-shaped.

The assumption here is that a document has one topic and lives in a region of semantic space. That is a real restriction — a document covering two subjects gets assigned to one cluster or to none — but it buys three things LDA cannot offer. Synonyms and paraphrases land together because the embedding puts them together, so short documents work far better. The number of clusters can be discovered rather than declared, since HDBSCAN infers it from density. And documents that belong nowhere are labelled as noise instead of being forced into a topic, which is usually the correct answer for a chunk of any real corpus.

The costs are honest too: you pay to embed every document, the pipeline has three stages each with parameters that interact, UMAP is stochastic so runs differ unless you fix a seed, and the topic labels are a post-hoc description of a cluster rather than something the model optimised.

The evaluation trap

The temptation is to pick by a number, and there is a published warning against the obvious one. Chang, Boyd-Graber, Gerrish, Wang and Blei, Reading Tea Leaves: How Humans Interpret Topic Models (NIPS 2009), had people perform word-intrusion and topic-intrusion tasks on fitted models and compared the results with held-out likelihood — the standard quantitative measure at the time. The relationship was often negative: models that scored better on predictive likelihood produced topics humans found less coherent.

That result is the reason the field moved to coherence measures such as NPMI, validated against human ratings by Lau, Newman and Baldwin (EACL 2014). It is also a general caution worth carrying beyond topic models: an unsupervised method optimises a proxy, and the proxy can move opposite to the thing you wanted. If the output is going to be read by a person, a human judgement of a sample is not a soft alternative to evaluation — it is the evaluation, and the number is the proxy.

Run both on your corpus

Nobody has run these on your documents and no published comparison predicts your corpus. Both fit in a few lines, so run them:

# --- LDA ---
from gensim import corpora
from gensim.models import LdaModel

texts = [[w for w in doc.lower().split() if w not in STOP and len(w) > 2]
         for doc in docs]
dictionary = corpora.Dictionary(texts)
dictionary.filter_extremes(no_below=5, no_above=0.4)   # the important knob
bow = [dictionary.doc2bow(t) for t in texts]
lda = LdaModel(bow, num_topics=20, id2word=dictionary,
               passes=10, random_state=0)
for i, topic in lda.print_topics(num_words=8):
    print(i, topic)

# --- embedding clustering ---
from sentence_transformers import SentenceTransformer
import umap, hdbscan

emb = SentenceTransformer("all-MiniLM-L6-v2").encode(docs,
                                                     show_progress_bar=True)
red = umap.UMAP(n_components=5, metric="cosine",
                random_state=0).fit_transform(emb)
labels = hdbscan.HDBSCAN(min_cluster_size=15).fit_predict(red)
print("clusters:", len(set(labels)) - (1 if -1 in labels else 0),
      "noise:", (labels == -1).sum())

Then judge them the way Chang et al. implies you should: for each method, print ten topics with their top terms, and for each topic pull three documents assigned to it. If you cannot name the topic from its terms, or the three documents do not belong together, the number of topics is wrong or the method is wrong for this corpus. That takes twenty minutes and settles it.

Choosing between them