Feature Engineering: Still the Highest-Leverage Work

“Feature engineering matters” is folklore until somebody says what a feature is. It is a change of coordinates: a way of re-describing the input so that the answer becomes reachable by the model class you are using. That statement can be proved on a problem small enough to check by hand.

What a feature actually buys

Every model class can express some functions and not others. A linear model can express weighted sums of its inputs and nothing else. A depth-3 tree can express any function that is constant on at most eight axis-aligned boxes. Fitting searches within that set; if the truth is outside it, no amount of data or tuning brings it in.

A feature moves the boundary of that set. It does not add information — every engineered column is a deterministic function of columns you already had, so in the information-theoretic sense you have gained nothing. What it does is put existing information in a form the model can use, which is a completely different thing and is the reason it works.

One column, proved

The classic impossibility. Two binary inputs, target is exclusive-or:

x₁  x₂   y
 0   0    0
 0   1    1
 1   0    1
 1   1    0

No linear model fits this. Proof: a linear model predicts w₁x₁ + w₂x₂ + b. Rows two and three require w₂ + b > 0 and w₁ + b > 0; adding them gives w₁ + w₂ + 2b > 0. Rows one and four require b ≤ 0 and w₁ + w₂ + b ≤ 0; adding those gives w₁ + w₂ + 2b ≤ 0. The two conclusions contradict, so no weights exist. This is the argument that stalled neural network research for a decade after Minsky and Papert made it in 1969.

Now add one engineered column, x₃ = x₁ · x₂, and the problem becomes trivially linear with w = (1, 1, −2), b = 0:

x₁ x₂ x₃    1·x₁ + 1·x₂ − 2·x₃    y
 0  0  0     0 + 0 − 0 = 0         0  ✓
 0  1  0     0 + 1 − 0 = 1         1  ✓
 1  0  0     1 + 0 − 0 = 1         1  ✓
 1  1  1     1 + 1 − 2 = 0         0  ✓

Exact, not approximate. One column turned an impossible problem into a solved one, and that is the mechanism behind every interaction term, every ratio, and every domain-expert suggestion that turns out to be worth more than a model upgrade.

Four transformations that keep paying

Ratios and differences

A linear model cannot divide. Given amount and account_balance it can weight each, but the quantity that predicts trouble is often amount / balance, and no combination of weights produces a quotient. The same applies to differences against a reference: price − category_median_price is usually far more predictive than either column alone. Tree models can approximate a ratio with enough splits, badly; give them the column.

Cyclical time

Hour of day encoded as 0–23 tells the model that 23:00 and 00:00 are twenty-three units apart, which is exactly wrong for anything with a daily rhythm. Map the circle onto a circle:

import numpy as np

df["hour_sin"] = np.sin(2 * np.pi * df["hour"] / 24)
df["hour_cos"] = np.cos(2 * np.pi * df["hour"] / 24)

# 23:00 and 00:00 are now adjacent:
#   hour 23 → (−0.259, +0.966)
#   hour  0 → ( 0.000, +1.000)

The same trick applies to day of week, month and angle. Recency — days_since_last_event — is the other time feature that earns its keep almost everywhere, and it is worth computing on a log scale, since the difference between 1 and 2 days matters more than between 300 and 301.

Aggregates over a group

Counts, means and standard deviations per entity: transactions per card in the last hour, average order value for this customer, number of distinct devices for this account. These are usually the strongest features in fraud, churn and abuse problems, and they are also the most dangerous — see the next section.

Categorical encodings

One-hot is safe and explodes in width. Ordinal encoding is compact and lies about order unless the order is real. Target encoding — replacing a category with the mean target for that category — is powerful, compact, and leaks the label directly into a feature unless it is computed out-of-fold. Use the library implementation that does the out-of-fold version, or write it yourself and test it.

A fifth family is easy to forget and cheap: explicit missingness. When a value is absent, the fact that it is absent is often itself predictive — an unfilled optional field, a sensor that dropped out, a customer who never supplied a phone number. Imputing the mean and moving on destroys that signal silently. Add a boolean was_missing column beside the imputed one and let the model decide whether it matters. Tree-based implementations handle missing values natively and will learn a direction for them, which is one more reason they need less preprocessing than a network does.

Where features leak

Three of the four families above can produce a column that would not have been available at prediction time, which makes the offline number fictional. The rules that prevent it:

• Ask when each input exists. For every column, write the timestamp at which its value is known. Any column whose timestamp is after the moment of prediction is disqualified, no matter how predictive it is. This single question catches most target leakage.
• Compute aggregates with a window that ends before the event. “Average order value for this customer” computed over the whole dataset includes the order you are predicting. Use a strictly prior window.
• Fit every encoder inside the fold. Target encoders, scalers, imputers and selectors all learn from the labels or the distribution, and fitting them before the split is the quiet leak that inflates a result by a point or two and is never noticed.

What deep learning did and did not change

For perceptual data the argument is over. Nobody hand-designs image filters or audio cepstral features any more, because learned representations comprehensively beat them, and nobody hand-designs text features because embeddings do it better. In those domains, feature engineering became architecture choice.

For tabular data it did not happen. The columns are already semantically meaningful, the relationships are irregular rather than smooth, and the winning approach remains good features plus a tree ensemble. The modern hybrid is worth knowing: embed the unstructured columns with a pretrained model, treat the resulting vector as a set of numeric features, and hand the whole table to the tree. You get learned representations where they help and engineered features where they still win.