3D Model Embeddings for Shape Similarity Search

A shape embedding turns a mesh or a scan into a fixed-length vector so that nearest neighbours in vector space are similar shapes. Everything difficult about it is in the word similar, which has at least four incompatible meanings and no default.

Decide what similar means first

Before choosing an encoder, decide which of these you want, because no single embedding gives all of them and they conflict.

• Same category. All chairs are near each other. This is a semantic notion and is what a classification-trained encoder gives you, because the training objective explicitly collapses within-category variation.
• Same geometry. This specific bracket, so you can find the duplicate part already in the catalogue. This is a retrieval notion and wants the opposite: fine geometric distinctions preserved, categories irrelevant.
• Same function. Things you can sit on, whatever they look like. Not recoverable from geometry alone and generally needs language supervision.
• Fits the same space. Bounding box, volume, clearances. Not an embedding problem at all — compute the numbers directly and filter on them, which is faster and exact.

A part-reuse search that returns “another bracket, but a different one” has not failed at retrieval; it has succeeded at the wrong objective. Fixing that is a training-data question, not a model-size question.

Multi-view: render it and use an image model

The multi-view approach sidesteps 3D architectures entirely. Render the object from a ring of virtual cameras, push each rendering through a standard 2D image network, and pool the per-view features into one descriptor. MVCNN, from Hang Su and colleagues at ICCV 2015, established this and used twelve views spaced 30° apart around the vertical axis, aggregated by an element-wise max across views.

The pooling is the same symmetric-function trick as in point cloud classification, applied across renderings rather than across points, and it earns the same property: the descriptor does not depend on the order the views were rendered in. The practical advantage is large and often decisive — it inherits image pretraining on hundreds of millions of photographs, which no 3D dataset comes close to matching in scale.

The costs are equally concrete. Rendering is required at query time, so an embedding takes tens of milliseconds rather than one forward pass. Interior structure invisible from outside is invisible to the descriptor, which matters for assemblies and for anything hollow. And a ring of cameras about the vertical axis assumes the object has a known upright orientation, which CAD libraries usually provide and raw scans usually do not.

Point-based and voxel encoders

The alternative consumes the geometry directly. Sample points from the surface — farthest point sampling to a fixed count, typically 1,024 or 2,048 — and run a set encoder whose pooled global feature is the embedding. This is what PointNet and its descendants produce before the classification head, and taking the pre-head vector is the standard way to get a shape descriptor out of a classifier.

Sparse voxel encoders do the same job with 3D convolution over occupied cells, which captures local geometry more naturally at the cost of the quantisation described on the voxel grid page. Both handle partial scans better than multi-view rendering does, because a scan with a missing back is a legitimate point set but an awkward thing to render from twelve angles.

Training matters more than architecture here. A classification loss produces an embedding that clusters by category and is nearly useless for finding a specific part. A metric loss — triplet, contrastive, or a self-supervised objective that treats two different samplings or crops of the same object as a positive pair — produces one where distance is graded rather than clumped, which is what retrieval needs.

Worked: ranking three candidates

With embeddings normalised to unit length, cosine similarity is just the dot product and the ranking is arithmetic you can check by hand. Four dimensions instead of the usual few hundred:

query   q = [0.65, 0.54, 0.43, 0.32]
cand.   a = [0.62, 0.58, 0.40, 0.34]
cand.   b = [0.10, 0.20, 0.95, 0.20]
cand.   c = [0.70, 0.50, 0.35, 0.37]

cos(q,a) = .65*.62 + .54*.58 + .43*.40 + .32*.34
         = .4030 + .3132 + .1720 + .1088 = 0.997

cos(q,b) = .0650 + .1080 + .4085 + .0640 = 0.646

cos(q,c) = .4550 + .2700 + .1505 + .1184 = 0.994

ranking: a (0.997), c (0.994), b (0.646)

note the gap. a and c are separated by 0.003 and b sits
0.35 away. that shape — a tight cluster and a far
outlier — is what a category-trained embedding produces,
and it is exactly why picking between a and c on this
score is not meaningful.

That last observation is the practical one. When the top candidates are separated by less than the noise in the embedding, the ranking among them is arbitrary, and the fix is a second stage rather than a better first one: retrieve the top fifty by vector search, then re-rank them with something exact and expensive — Chamfer distance after rigid alignment, or a direct geometric comparison of the mesh. Retrieval finds the neighbourhood; geometry picks the winner.

The invariances you must choose

• Rotation. Neither family is rotation invariant by construction. Multi-view around a vertical axis is invariant to vertical-axis rotation only, and only because of the max pool. Point encoders are not invariant to anything unless trained with augmentation. If your data has a reliable upright direction — CAD libraries and building scans usually do — exploit it; it is far cheaper than making the model handle arbitrary rotation.
• Scale. Normalising into a unit sphere is standard and throws away absolute size, so a model aircraft and an airliner become the same vector. If size distinguishes your classes, keep it as an explicit metadata filter rather than hoping the embedding retains it.
• Sampling density and mesh tessellation. The same object exported with different triangle counts must not give different vectors. Sampling points uniformly by surface area rather than by vertex fixes the mesh-density problem, and is a preprocessing detail that quietly determines whether the system works at all.
• Symmetry. A left-hand and a right-hand part are mirror images. Many pipelines augment with reflections, which makes the two indistinguishable — a catastrophic outcome in a parts catalogue and a desirable one in a furniture search. Decide, do not inherit.

Indexing, and the domain gap that ruins it

Once shapes are vectors, retrieval is ordinary vector search: an approximate nearest neighbour index over a few hundred dimensions, with the same trade between recall and latency as any other embedding workload. The general treatment applies unchanged — see how HNSW works, which similarity metric to use and what the dimension count buys you. Normalise before indexing if you intend to use cosine similarity, or the index is answering a different question than you think.

The 3D-specific failure is the domain gap between clean models and real scans. Public benchmarks are built on CAD collections — ShapeNet, whose core subset holds tens of thousands of models across dozens of categories, and ModelNet40 — and those are watertight, complete, untextured and perfectly upright. The ShapeNet paper describes the collection and its annotations. A scan is none of those: it has holes where the scanner could not reach, noise, a floor still attached, and an arbitrary orientation. An encoder trained only on CAD models embeds a scan of a chair somewhere unhelpful, and the symptom is a retrieval system that works beautifully on the demo library and collapses on real data.

The mitigations are unglamorous and effective: train on simulated scans — render depth maps from the CAD models, add realistic noise and occlusion, and reconstruct clouds from those — or clean the query instead, removing the ground plane and cropping to the object before embedding. Which is cheaper depends on whether you control the query pipeline, and if you are building an as-built model anyway, the cleaning is already happening; see building a digital twin from a 3D scan.