Machine Learning With 400 Rows

With 400 rows the binding constraint is not which algorithm you pick. It is that the dataset can support about six features and that your test set cannot tell an 80% model from an 88% one. Both numbers are derivable in a line, and both settle arguments that otherwise run for weeks.

How many features 400 rows can pay for

The relevant quantity is not the row count. It is the number of events — occurrences of the rarer class — because that is what the model has to learn from.

400 rows, 15% positive  ->  60 events

A long-standing convention in the biostatistics literature is 10 to 20
events per predictor variable for a logistic regression to produce
stable, non-optimistic coefficients.

  60 events / 10 per predictor  =  6 predictors
  60 events / 20 per predictor  =  3 predictors

Not six after selection from fifty. Six chosen before looking at the
data, because choosing them by looking is itself a use of the events
and is the leakage pattern in the selection section of the leakage page.

Six. That is the whole budget, and it is why most small-data projects are really feature-selection projects disguised as modelling projects. Every additional column past the budget buys a small amount of apparent fit and a large amount of variance.

Interactions and polynomial terms count against the budget too. So does a one-hot encoding: a categorical with eight levels is seven predictors, which on 60 events consumes the entire allowance for one column. Bin it to two or three levels using domain knowledge, before you look at the outcome.

What your test score actually says

Split 400 rows 80/20 and the test set is 80 rows. The standard error of an accuracy estimated on n rows is the standard error of a proportion.

SE = sqrt( p * (1 - p) / n )

n = 80, observed accuracy 0.80:
  SE = sqrt(0.80 * 0.20 / 80) = sqrt(0.002) = 0.0447
  95% interval = 0.80 +/- 1.96(0.0447) = 0.80 +/- 0.088
               = 0.712 to 0.888

So a model reported at 0.80 and a model reported at 0.88 are not
distinguishable on this test set. Neither is 0.72.

How large would the test set need to be for a +/- 0.05 interval?

  1.96 * sqrt(0.8 * 0.2 / n) < 0.05
  sqrt(0.16 / n) < 0.0255
  0.16 / n < 0.00065
  n > 246

You would need 246 test rows -- more than half your entire dataset --
to state accuracy to within five points.

This one calculation ends most small-data disputes. The team that spent three weeks moving accuracy from 0.79 to 0.83 did not measure an improvement; they measured noise, and they will find out when the model meets new data. Report the interval in every result, and the argument about which model won stops happening.

Evaluating without a test set you can spare

1. Use repeated stratified k-fold, not a single holdout. Ten folds repeated ten times gives 100 estimates and every row is used for both training and evaluation. The spread across repeats is the number to report, and it is usually alarming.
2. Nest the tuning. If you select hyperparameters using cross-validation and then report that same cross-validation score, the score is optimistic by an amount that grows with how many configurations you tried. Nested cross-validation — an inner loop for tuning, an outer loop for estimating — is expensive and, at 400 rows, cheap in absolute terms.
3. Prefer 10-fold over leave-one-out. Leave-one-out is nearly unbiased and has high variance, and its folds are almost identical to each other, so the estimates are highly correlated and the apparent precision is illusory.
4. Stratify everything. With 60 positives, an unstratified 10-fold split will produce folds containing two positives, and a fold with two positives produces a meaningless score.
5. Keep one genuinely untouched holdout if you can afford it. Forty rows tells you almost nothing about the level, but it will catch a catastrophic failure, and it is the only defence against having tuned against the cross-validation a hundred times.

Models that behave at this size

• Penalised logistic regression. The default, and hard to beat. L2 shrinks all coefficients and handles correlated features; L1 selects, which at this size is a way of spending your budget automatically rather than by hand. Tune the penalty by nested cross-validation and expect a strong one.
• A shallow tree, depth 2 or 3. Not for accuracy but for legibility: at 400 rows a tree the domain expert can read is worth more than a point of AUC, because the expert will spot the split that encodes a data-collection artefact.
• A regularised random forest. Set min_samples_leaf to 10 or 20 rather than 1. It cannot overfit by adding trees, which removes one whole class of mistake — see the variance argument.
• Gradient boosting with a very small learning rate and heavy constraints. Possible, and rarely worth it. Boosting has enough capacity to memorise 400 rows in a handful of trees, and the settings that prevent it mostly turn it back into a linear model.
• Not a neural network. There is nothing here for it to learn a representation from, and there is nothing to pretrain on, which is the same argument as why trees win on tables generally, only sharper.

Putting knowledge in where data is missing

Small data is exactly the regime where prior knowledge earns its keep, because there is not enough evidence to overwhelm it and no reason to pretend otherwise.

• Monotonic constraints. If you know that risk rises with the value of a feature, say so. The model stops needing to learn it from 60 events, and it stops learning the reversal that a thin slice of data suggested. The credit-scoring page shows the parameter.
• Composite features. Combine several weak columns into one meaningful ratio using domain knowledge — utilisation rather than balance and limit separately. One informative feature costs one unit of budget where three raw ones cost three.
• Partial pooling. Where the data has groups — branches, clinicians, regions — a hierarchical model shrinks each group’s estimate towards the overall mean by an amount decided by how much data that group has. It is the principled version of what the smoothing term in target encoding does by hand.
• Informative priors, stated. A Bayesian model lets you write down what is known before the data and report a posterior that combines both. The obligation is to publish the prior, so a reader can judge how much of the conclusion came from evidence.
• Transfer from a related dataset, carefully. A model fitted on a larger neighbouring population, then adjusted, can beat anything fitted on 400 rows alone. The assumption — that the relationship transfers — is exactly the kind that needs stating and checking.

When to stop and write a rule

With 400 rows, the honest comparison is not model against model. It is model against the two-line heuristic the domain expert already uses.

1. Write down the expert’s current rule. Ask what they check. It will be two or three conditions and it will be surprisingly good.
2. Score it on the same cross-validation as the model. Same folds, same metric, same interval. This step is skipped almost universally and it is the only fair comparison available.
3. Compare the intervals, not the point estimates. If they overlap substantially, the model has not been shown to be better, and it costs pipelines, monitoring, retraining and an on-call rota that the rule does not.
4. Consider a fitted rule as the middle option. A depth-2 decision tree is a rule with thresholds chosen from data. It keeps almost all of the interpretability and typically recovers most of the gain over the expert’s version.
5. If the model wins, ship it with the rule as a guardrail. Log every case where the two disagree and review them. On a small dataset, disagreement is where the model is extrapolating, and that is where it will be wrong first.

The wider version of that decision — when a rule, a heuristic or a SQL query is simply the correct answer — belongs to the theory cluster, and at 400 rows it applies more often than anyone building the model wants to hear.