Causal Inference When All You Have Is Observational Data

Predictive modelling asks what happens next. Causal inference asks what would happen if you intervened, and no amount of predictive accuracy answers it. The gap is not a modelling gap — it is an assumption gap, and the assumptions have to be written down before the data is touched.

A different question, not a harder model

Write the two quantities and the difference is structural.

prediction:  E[ Y | X = x ]
             the average outcome among units that HAVE x

causation:   E[ Y | do(X = x) ]
             the average outcome if we SET x for everyone

The first is a fact about your dataset and is estimable from it. The second is a fact about a world that did not happen, and no dataset contains it. Bridging the two requires assumptions about how units came to have the values they have — assumptions that cannot be tested from the same data, which is why they must be stated.

A randomised experiment closes the gap by construction: assignment is independent of everything, so the two quantities coincide. That is why running the experiment beats every technique on this page, and why the techniques exist only for when you cannot.

A reversal, worked

A B2B product. Does an onboarding call improve 12-month retention? Invented counts, chosen so the arithmetic is checkable.

AGGREGATE

                 customers   retained   rate
  had a call        1,000        920    92.0%
  no call           1,000        744    74.4%
                                        ------
  naive effect                          +17.6 pp

SPLIT BY COMPANY SIZE

  small companies
    had a call        200        160    80.0%
    no call           800        560    70.0%      difference +10.0 pp

  large companies
    had a call        800        760    95.0%
    no call           200        184    92.0%      difference  +3.0 pp

  (check: 160 + 760 = 920 and 560 + 184 = 744)

STANDARDISED TO THE POPULATION (1,000 small, 1,000 large)

  adjusted effect = 0.5(10.0) + 0.5(3.0) = 6.5 pp

  naive     +17.6 pp
  adjusted   +6.5 pp
  the confounder accounted for 11.1 of the 17.6 points

Company size is a confounder: it causes both the treatment (the sales team prioritised large accounts for calls) and the outcome (large accounts retain better anyway). The naive comparison credits the call with a difference that was mostly a difference in who got called.

A predictive model would have found the same thing and called it a feature. “Had an onboarding call” is genuinely predictive of retention — it is a proxy for being a large account. It is only when somebody proposes to call everyone that the distinction becomes expensive, because the intervention does not carry the account-size effect with it.

With a strong enough confounder the effect can reverse entirely, so that the treatment appears better in aggregate and worse in every subgroup. That is Simpson’s paradox, and the arithmetic above is the mild version of it.

Drawing the assumptions down

A directed acyclic graph is a picture of what you believe causes what. It is not fitted from data; it is written from domain knowledge, and its value is that it makes the belief arguable.

             company_size
              /          \
             v            v
    onboarding_call --> retention

  company_size is a confounder: it points into both.
  Adjusting for it CLOSES the back-door path
    onboarding_call <- company_size -> retention
  and what remains is the direct effect.

The backdoor criterion makes this precise. To identify the effect of X on Y, block every path from X to Y that starts with an arrow pointing into X, and do not open any new ones. Practically: adjust for common causes of treatment and outcome, and for nothing that lies on the causal path between them.

That last clause matters. If the onboarding call works by getting the customer to connect an integration, then adjusting for “connected an integration” removes exactly the effect you were trying to measure. Adjusting for a mediator gives you the effect that does not run through it, which is a different quantity and is almost never the one anyone wanted.

A DAG is also the right place to notice that the outcome has timing. “Retained at 12 months” collapses a duration into a boolean, and where the treatment shifts when people leave rather than whether, a survival model over the same covariates answers the question the binary comparison cannot. The identification argument is identical; only the outcome changes.

The variable you must not adjust for

The most damaging error in applied causal work is not omitting a confounder — it is adjusting for a collider, because that manufactures an association where none existed.

  product_fit ---> upgraded <--- heavy_usage

  upgraded is a COLLIDER: two arrows point into it.

Suppose fit and usage are entirely independent in the population,
and either one alone is enough to trigger an upgrade. Now look only
at customers who upgraded:

  a customer who upgraded with poor product fit almost certainly
  had heavy usage -- otherwise they would not have upgraded

Among upgraders, fit and usage are now negatively correlated. The
correlation is created by the selection, not present in the world.

This appears in production analysis constantly and is almost never recognised. “Among our paying customers”, “among users who completed onboarding”, “among tickets that reached tier 2” — every one of those is conditioning on a variable that multiple causes point into, and every one can generate a confident finding about a relationship that does not exist.

The practical rule is uncomfortable and correct: you cannot decide what to adjust for by looking at the data. Adding a variable can remove bias or create it, and which one it did depends entirely on the causal structure you believe. “Control for everything available” is not a conservative choice; it is an unexamined one.

Four designs and what each one buys

Look for the natural experiment before reaching for the adjustment. A staged rollout, a capacity constraint, a system outage that suppressed a treatment for a week, an eligibility threshold — these produce quasi-random variation and are worth more than any amount of covariate adjustment.

How wrong could you be

Since the central assumption is untestable, the honest response is to quantify what it would take to overturn the finding rather than to assert it holds.

1. Ask how strong an unmeasured confounder would have to be. Sensitivity analysis expresses the finding as a threshold: an unobserved variable would need to be associated with both treatment and outcome by at least some factor to explain the whole effect. If that factor is implausibly large given what you already measured, the finding is robust; if it is the size of a variable you know exists and did not record, it is not.
2. Run a negative control. Pick an outcome the treatment could not possibly affect and estimate the effect on it. A non-zero result there measures your residual confounding directly. This is cheap and is the single most persuasive check available.
3. Vary the specification. Report the estimate under several defensible adjustment sets. A finding that moves from +6.5 to +1.2 depending on which covariates are included is not a finding, and reporting only the specification you liked is the observational equivalent of slicing until significant.
4. Check overlap. If no large accounts went uncalled, no method estimates the effect for large accounts; it extrapolates. Plot the propensity distribution by arm and restrict to the region where both exist, then say that you did.

What you may honestly write

The output of an observational analysis is a sentence, and the sentence has a required shape. Not this:

“Onboarding calls increase 12-month retention by 6.5 percentage points.”

This:

“Among customers observed between January 2025 and June 2026, those who received an onboarding call retained 6.5 percentage points more often than those who did not, after adjusting for company size, plan tier and acquisition channel. This estimate assumes no unmeasured factor drives both the decision to call and retention; the sales team selected accounts partly on signals we did not record, so that assumption is doubtful. An unmeasured confounder roughly as strong as company size would remove the effect entirely. A randomised trial on the mid-market segment would settle it, and we estimate it needs 4,000 accounts over four months.”

The second is longer, weaker-sounding, and the only one that is true. It also does the more useful thing: it names the experiment that would resolve the question, which is what turns an analysis into a decision. Where the intervention will be targeted rather than universal, the uplift framing is the version of this that produces a list, and it needs the same randomisation to be worth anything.