Clustering Embeddings for Topic Discovery

You have 200,000 support tickets and no taxonomy. Reading a sample gives you the loud categories and misses the long tail, which is where the interesting failures live. Clustering embeddings is the standard answer, and the standard implementation of it has two steps most write-ups skip.

What clustering is actually for

The output is not clusters. The output is a list of named themes with counts and examples, which is a thing a product manager can act on. Everything below is in service of getting to that, and a pipeline that produces 340 unnamed integer cluster ids has done maybe a third of the work.

The tasks this genuinely serves: discovering support categories you did not know existed, finding the topics behind a spike in volume, building the initial label set for a classifier you will later train, and auditing a corpus before you index it. It is exploratory analysis, and the standard for success is that a human reads the result and learns something.

Why k-means is the wrong first choice

k-means is the default in every tutorial and it fights the data in three ways. It requires you to state k, which is the thing you are trying to discover. It assigns every point to a cluster, so genuine one-off tickets get forced into whichever theme is least wrong and quietly pollute it. And it finds spherical clusters of similar size, which is not what embedding space contains — real topic distributions are wildly unequal, with one enormous “password reset” blob and a hundred small specific ones.

HDBSCAN (Campello, Moulavi and Sander, 2013) fixes all three. It infers the number of clusters from the density structure, it allows clusters of different shapes and sizes, and — most valuable here — it labels points that belong to no cluster as noise rather than forcing them. That noise set is often 20 to 40% of a real corpus and it is not a failure; it is the model correctly telling you those tickets are not part of a recurring theme.

The pipeline that works

This is the shape BERTopic (Grootendorst, 2022) popularised, and it is worth understanding as four independent stages rather than as a library:

import umap, hdbscan, numpy as np

X = np.load("embeddings.npy")            # (200000, 1536), normalised

# 1. reduce - density-based clustering does not work at 1536 dims
Z = umap.UMAP(
        n_components=5,
        n_neighbors=15,
        min_dist=0.0,
        metric="cosine",
        random_state=42,
    ).fit_transform(X)

# 2. cluster
labels = hdbscan.HDBSCAN(
        min_cluster_size=50,
        min_samples=10,
        metric="euclidean",
        cluster_selection_method="eom",
    ).fit_predict(Z)

# labels == -1 is noise, and that is a feature

The reduction step is not an optimisation, it is a correctness fix. Density-based clustering depends on distances being informative, and in 1536 dimensions distances between points concentrate — the nearest and farthest neighbours of a point end up nearly equidistant, so “dense region” stops meaning anything. Projecting to five or ten dimensions restores the contrast HDBSCAN needs. Note also min_dist=0.0: you are reducing for clustering, not for a picture, and you want points packed as tightly as the structure allows. Use a separate two-dimensional UMAP if you also want to plot it.

The parameters that matter

Set random_state on UMAP. It is stochastic, and without a fixed seed you cannot tell whether a change in your clusters came from your parameter change or from the coin flips. Be aware that fixing the seed disables UMAP’s parallelism and slows the fit — worth it while you are iterating.

The labelling pass

Two approaches, and the good pipelines use both. The cheap one is class-based TF-IDF: treat all documents in a cluster as one large document and score terms by how characteristic they are of that cluster relative to the rest of the corpus. This costs nothing, runs in seconds, and produces a keyword list rather than a name — refund, charged, twice, duplicate — which is usually enough for an analyst and not enough for a dashboard.

The readable one is to have a language model name each cluster from its most central members. The cost is small and computable: for 200 clusters, send the 10 documents nearest each centroid — say 200 tokens each — and ask for a five-word label. That is 200 clusters × 2,000 tokens ≈ 400,000 input tokens plus a few thousand output tokens, one batch, a handful of cents at typical rates. Pick the members by distance to the centroid rather than at random; the central examples are what the cluster is about, and a random sample includes its boundary cases.

Pitfalls

• Clusters by format, not by topic. If every ticket begins with the same auto-generated header, the embeddings encode the header and your top-level split will be by template. Strip boilerplate before embedding; this is the most common cause of a result that looks structured and says nothing.
• Length as a hidden dimension. Very short and very long documents separate on length alone. Bucket by length and check whether your clusters merely recovered the buckets.
• Using retrieval prefixes. If your model has a clustering input type, use it. Query-mode embeddings are shaped for a different geometry.
• Treating noise as failure. A third of points unassigned is normal and informative. Sample the noise set and read it — it is where the genuinely novel items are.
• Re-running with new data and comparing ids. Cluster ids are not stable across runs. If you need to track a theme over time, match clusters between runs by centroid similarity, or freeze the labels and turn the problem into classification.