Credit and Risk Scoring Under Regulation

A credit model is not judged only on its discrimination. It has to survive a question no other production model faces: why was this particular person declined, answered in terms they can act on, months after the decision, by someone who was not there. That requirement shapes the model from the first design choice onward.

Why this model class is different

In most machine learning, an unexplainable model that scores better wins. Here it does not, for three reasons that are all structural rather than cultural.

• An adverse decision has to be explainable per applicant. Not the model in general — this decision, in a small number of specific factors, in language a customer can act on. That is a per-row requirement, and it rules out anything whose output cannot be attributed back to inputs.
• The decision has to be reproducible later. A dispute arrives eighteen months after the application. You need the exact model version, the exact feature values as they were at the time, and the exact threshold — which makes model and feature versioning a compliance artefact rather than an engineering nicety. Link this to point-in-time feature retrieval and to dataset versioning.
• The model must not discriminate on protected characteristics, including through proxies. Removing the protected column is not sufficient and is the most common misunderstanding in the field. See below.

The scorecard, and why it survived

The traditional credit scorecard is a logistic regression on binned features, with each bin’s contribution converted into points. It has been the industry default for decades, and the reason is not inertia: every property above falls out of it for free.

Binning, then weight of evidence, then logistic regression.

For a bin b of some feature:

  WoE(b) = ln( share of GOODS in b  /  share of BADS in b )

  age 18-24    goods 6%   bads 14%   WoE = ln(0.06/0.14) = -0.847
  age 25-34    goods 22%  bads 26%   WoE = ln(0.22/0.26) = -0.167
  age 35-49    goods 38%  bads 34%   WoE = ln(0.38/0.34) = +0.111
  age 50+      goods 34%  bads 26%   WoE = ln(0.34/0.26) = +0.268

Information value of the feature:

  IV = sum over bins of (share_goods - share_bads) * WoE
     = (0.06-0.14)(-0.847) + (0.22-0.26)(-0.167)
       + (0.38-0.34)(0.111) + (0.34-0.26)(0.268)
     = 0.0678 + 0.0067 + 0.0044 + 0.0214
     = 0.100

Conventional reading of IV: below 0.02 useless, 0.02-0.1 weak,
0.1-0.3 medium, 0.3-0.5 strong, above 0.5 suspicious -- check for
leakage before celebrating.

The binning is what buys the properties. It makes the model a lookup table, so any decision decomposes exactly into per-feature point contributions; it makes the relationship monotone by design where you need it to be; it makes outliers and missing values into ordinary bins rather than special cases; and it makes the whole thing stable when the input distribution shifts a little, because a shift within a bin changes nothing.

The cost is discrimination. A well-tuned gradient-boosted model will usually beat a scorecard on a held-out sample. Whether the difference is worth the explainability work is the actual decision, and it depends on the product — a high-volume unsecured product where a point of Gini is worth a great deal argues differently from a low-volume one.

Monotonic constraints: keeping boosting sane

If you do use boosting, constrain it. All three major implementations support monotonic constraints, which force the model’s output to move in one direction as a feature increases.

# LightGBM: one entry per feature, in feature order
#   1 = non-decreasing, -1 = non-increasing, 0 = unconstrained
params = {
    "objective": "binary",
    "monotone_constraints": [1, -1, 0, 1],
    "learning_rate": 0.03,
    "num_leaves": 15,
}

# XGBoost accepts the same idea as a tuple or a string
#   monotone_constraints=(1, -1, 0, 1)

This is not only about defensibility. An unconstrained booster on real credit data routinely learns that risk falls somewhere in the middle of the income range, because a thin slice of the training data happened to behave that way. That relationship is noise, it will not hold next year, and it is indefensible in a review. Constraining it away typically costs a trivial amount of held-out performance and removes an entire class of embarrassment.

Producing reasons a person can act on

A reason code is not a feature importance. Importance tells you what matters to the model on average; a reason code has to say what pushed this application below the line, and it has to be stated as something the applicant could plausibly change.

For a scorecard the computation is exact and needs no library. For each feature, take the difference between the points this applicant scored and the points they would have scored in the best-performing bin. Sort descending. Report the top three or four.

applicant points by feature      best-bin points      shortfall
  months_at_address    12               40                  28   <- 1
  revolving_utilisation 18              45                  27   <- 2
  recent_enquiries      22              38                  16   <- 3
  age_of_oldest_account 41              44                   3
  income_band           35              36                   1

reasons reported:
  1. length of time at current address
  2. proportion of available revolving credit in use
  3. number of recent credit searches

For a boosted model, SHAP values give the per-row decomposition and are the standard tool, with the caveat that they explain the model rather than the world — what an explanation is and is not evidence of is the page that matters before anyone puts a SHAP output in front of a regulator. Two practical requirements: the explanation must be produced from the same model version that made the decision and stored with it, and the phrasing must be pre-approved text rather than a feature name rendered raw. “f_util_ratio_3m was high” is not a reason anyone can act on.

Protected characteristics and their proxies

Deleting the protected column does not remove the effect. Postcode correlates with ethnicity in most countries. Job title, working hours and career-gap patterns correlate with sex. Device type, name string handling and even the hour of application correlate with things you did not intend to model. A sufficiently good model reconstructs the removed attribute from what is left, and does so more effectively the more features you give it.

1. Retain the protected attribute for testing, separately from the modelling table. You cannot measure disparate impact on a characteristic you did not record. Storing it in a controlled place for measurement only, never joined into the feature set, is the standard arrangement — and is itself subject to data-protection rules worth checking.
2. Measure outcome rates by group. Approval rate, average score, and — crucially — default rate among those approved. A model can be unfair in ways that are invisible in approval rates alone.
3. Test each feature for proxy strength. Fit a model that predicts the protected attribute from the candidate feature set. Whatever it can predict well, your risk model can also use. Features that individually predict a protected characteristic strongly need a documented business justification or removal.
4. Decide what fairness definition you are using, and write it down. Equal approval rates, equal false-positive rates and equal calibration across groups are mutually incompatible except in degenerate cases — this is a mathematical result, not a policy disagreement. You must pick, and the choice belongs to compliance and the business rather than to the modeller.

The training set you cannot observe

Here is the structural problem unique to lending. You only observe repayment behaviour for applicants you approved. The declined ones have no outcome, so a model trained on the approved population is trained on a sample selected by the previous model — and it learns that whatever the old policy rejected is unmeasurable rather than bad.

The standard responses are reject inference (imputing outcomes for declined applicants, with all the assumptions that implies) and, far more informative, a small randomised approval holdout: approve a small random fraction of marginal applicants regardless of score, and observe them. That deliberate loss buys the only unbiased data anyone will ever have about the region near the cut-off. It is the same instinct as keeping a randomised control group in a campaign, and it is expensive for the same reason and worth it for the same reason.

Population stability, worked

The population stability index measures how far the current distribution of scores has moved from the one the model was built on. It is the standard monitoring number in this field, it needs no labels, and it is one line of arithmetic per band.

PSI = sum over bands of  (actual% - expected%) * ln(actual% / expected%)

band   expected   actual   (a - e)    ln(a/e)     contribution
  1      0.20      0.30     0.10      0.4055        0.04055
  2      0.20      0.22     0.02      0.0953        0.00191
  3      0.20      0.18    -0.02     -0.1054        0.00211
  4      0.20      0.18    -0.02     -0.1054        0.00211
  5      0.20      0.12    -0.08     -0.5108        0.04087
                                             PSI =  0.0875

Conventional bands (industry convention, not a regulation):
  PSI < 0.10          stable
  0.10 <= PSI < 0.25  investigate
  PSI >= 0.25         material shift; treat the model as suspect

At 0.0875 this population is inside the stable band but the shape of the movement is informative on its own: applicants are piling into the lowest score band and draining out of the highest. That is a change in who is applying, not a change in the model, and it will hit approval volumes and loss rates before any PSI threshold trips. Look at the contributions, not only the total.

Run PSI on the score, on every input feature, and on the approval rate, monthly. The general version of this monitoring — including the difference between the inputs moving and the relationship changing — is the model drift page.

What has to be written down

Model documentation in this field is not a README. It is a controlled document, usually reviewed by an independent validation function, and the engineering job is to make its contents derivable from the repository rather than retyped.

Broader regulatory context sits outside this cluster: the EU AI Act, algorithmic accountability and AI in financial services each cover a part of it.