Ordinal and Nominal Categorical Features: Why the Distinction Changes the Encoding

“Small, medium, large” and “red, green, blue” look identical in a CSV: both are three strings in a column. One of them carries an order that a model can exploit and the other does not, and the two ways of getting that wrong fail in opposite directions.

The distinction, precisely

A feature is ordinal if its categories have a total order that is meaningful for the outcome: education level, severity grade, star rating, T-shirt size, a survey’s Likert scale, the bins from discretising a continuous column. It is nominal if the categories are merely different: country, payment method, product SKU, colour.

The critical qualifier is “meaningful for the outcome”. An order that exists in the world but not in the relationship you are modelling is not usable order. Months have a calendar order, but if your target is seasonal, December is adjacent to January and the integer encoding 1…12 asserts they are eleven apart. Postal codes sort, but the sort is not a distance. Treating either as ordinal imposes a false geometry.

Equally, ordinal features do not have known spacing. Ordinal means “small < medium < large”, not “medium is exactly halfway”. Encoding as 0, 1, 2 asserts equal gaps, which is a stronger claim than the data supports — and it matters for a linear model, which will fit one coefficient to that spacing, while it does not matter for a tree, which only ever compares against thresholds.

What a tree does with each

A decision tree on a numerically encoded feature considers splits of the form x <= t. If the encoding is ordered, every such split is a contiguous cut: “small and medium” versus “large”. There are exactly k − 1 such cuts for k categories, every one of them meaningful, and the tree can reach any contiguous grouping by combining two of them at successive depths.

A one-hot encoded feature gives the tree k binary columns, and each split is “is it this one category, or not”. To express “large or extra-large” the tree needs two levels of depth and two separate splits; to express “at least medium” on a five-level scale it needs three. Each of those splits is estimated from a smaller and smaller subset of rows, so each is noisier, and depth spent on reassembling an order is depth not spent on interactions with other features.

A worked cost in splits

Take a five-level severity feature — minimal, mild, moderate, severe, critical — where the true effect is monotone: escalation probability rises steadily with severity. Suppose the useful rule is “severe or critical escalate”.

Ordinal encoding (0…4): one split, severity > 2.5. It is estimated from all the rows, it is the first split the tree finds because it maximises the gain, and it generalises to a category the model saw rarely, because a rarecritical row still falls on the correct side of the threshold learned mostly from severe rows.

One-hot encoding: two splits at two depths — is_severe == 1, then within the negative branch is_critical == 1. Both are found only if both categories are frequent enough for the gain to clear the minimum-child-weight threshold. Suppose critical is 0.5% of rows: on 20,000 rows that is 100 rows, and after two levels of prior splitting, the node containing them may hold twenty. A min_child_weight of 25, which is an unremarkable setting, prevents that split existing at all. The model does not predict critical badly — it cannot see it, and it will predict for a critical row whatever it predicts for “none of the other four”.

import numpy as np, pandas as pd
from sklearn.model_selection import cross_val_score, StratifiedKFold
from sklearn.ensemble import HistGradientBoostingClassifier

rng = np.random.default_rng(0)
levels = ["minimal", "mild", "moderate", "severe", "critical"]
weights = [0.34, 0.30, 0.24, 0.115, 0.005]          # critical is rare
sev = rng.choice(levels, size=20_000, p=weights)

rank = {name: i for i, name in enumerate(levels)}
risk = np.array([0.02, 0.05, 0.11, 0.42, 0.61])     # monotone in rank
y = rng.binomial(1, risk[[rank[s] for s in sev]])

ordinal = np.array([[rank[s]] for s in sev])
onehot  = pd.get_dummies(pd.Series(sev)).to_numpy()

cv = StratifiedKFold(5, shuffle=True, random_state=0)
clf = HistGradientBoostingClassifier(min_samples_leaf=25, random_state=0)
print("ordinal:", cross_val_score(clf, ordinal, y, cv=cv, scoring="roc_auc").mean().round(4))
print("one-hot:", cross_val_score(clf, onehot,  y, cv=cv, scoring="roc_auc").mean().round(4))

The gap widens as the rare level gets rarer and as min_samples_leaf rises, which is the mechanism stated back as a knob: the ordinal encoding shares statistical strength across adjacent levels and the one-hot encoding does not. On a balanced five-level feature with plenty of rows, the two converge and the choice stops mattering much for a tree — it is the rare tail where the information is lost.

The opposite mistake, which is worse

Encoding a nominal feature as an integer is the more damaging error, because it does not merely withhold information — it asserts something false. Give a linear model country = 7 and it will fit a coefficient claiming that country 8 is one unit more than country 7 in the direction of the target, and that country 14 is twice as far from country 0 as country 7 is. There is no such quantity.

For trees the harm is subtler and still real. A tree can recover arbitrary groupings of an integer-coded nominal feature, but only by spending depth on contiguous cuts in an ordering that is arbitrary. If the true grouping is {3, 11, 19} versus the rest, a tree must isolate each with its own pair of thresholds. Depth is finite, so with enough categories the grouping simply is not found.

There is one exception that is genuinely useful rather than a mistake: a boosted tree given an integer-coded high-cardinality nominal feature often performs acceptably anyway, because with enough trees and enough depth the ensemble can carve out the groupings it needs across many trees even if no single tree can. It is not free — it costs trees and depth that would otherwise model something else — but it explains why the practice survives despite being wrong in principle.

Worse, the arbitrary ordering is usually alphabetical, because that is what OrdinalEncoder does by default when you do not pass categories. A model whose splits depend on the alphabet is a model whose behaviour changes when a category is renamed.

Choosing an encoding

• Ordinal, order known: integer codes in the correct order, set explicitly — OrdinalEncoder(categories=[["minimal", ...]]). Never rely on the default sort.
• Ordinal, unequal spacing that matters, linear model: integer codes plus a spline or a set of threshold dummies (“at least medium”, “at least severe”), which keeps the order while letting each step have its own coefficient.
• Nominal, low cardinality: one-hot, or the native categorical support in LightGBM and CatBoost, which partition categories at a node without expanding columns.
• Nominal, high cardinality: target encoding with cross-fitting, or hashing. Both are covered in categorical encoding methods. Target encoding without cross-fitting is a leak, not an encoding.

The one thing not to do is let a type detector decide for you. Whether a column is ordinal is a fact about meaning, not about dtype, and no amount of inspecting the values reveals it — automatic column type detection can tell you a column holds five distinct strings and cannot tell you whether they have an order.