Data Analysis With LLMs: Where It Breaks

Ask a model to analyse a dataset and it writes code, the code runs, real numbers come out, and a paragraph explains what they mean. Three independent things had to be right. Only one of them tells you when it was not.

Three places to be wrong

The code can be wrong. If it crashes you find out immediately, which is the benign case. The dangerous case is code that runs cleanly and computes something other than what you asked.

The statistics can be wrong. The code faithfully executes a procedure whose assumptions the data violates, or which answers a different question from the one you have. Nothing errors; the number is simply not evidence for what you think.

The interpretation can be wrong. This is where the model is on its home turf and at its most dangerous, because generating a fluent explanation of a result is exactly what it is good at, and it will do so with equal confidence whether the result supports the explanation or not.

Code that runs and is wrong

A short list of things that produce no error and change the answer. Every one of them is ordinary and none is specific to models — but a human writing the code usually knows the dataset, and the model does not.

• Silent row loss. Missing values dropped by default somewhere in the chain, so the analysis runs on a subset that is not random with respect to the outcome.
• Joins that change cardinality. A merge intended as one-to-one that is actually many-to-many, silently duplicating rows and inflating every count and every significance test downstream.
• Type coercion. A column read as text because of one stray value, then coerced to numbers with the failures becoming missing values that get dropped by the previous bullet.
• Grouping that discards keys. Missing group labels dropped by default, so an entire category disappears from a breakdown without appearing anywhere in the output.
• Units and encodings. A column the model assumed was a percentage and is a proportion; a sentinel value like -999 treated as a measurement; a date parsed with the wrong day-month order for the first twelve days of each month.

The single highest-yield habit against all of these is to demand row counts. Print the number of rows before and after every filter, join and aggregation, and read them. Most of the list becomes visible immediately, and it costs one line per step.

The statistical errors

• Multiple comparisons. Ask for “anything interesting” in a wide table and the model will test many things and report what came out significant. That is the textbook recipe for a false positive, and no correction is applied unless you ask.
• Violated independence. The most common serious error. Repeated measures on the same subject, multiple samples per site, autocorrelated time series — all analysed with tests that assume independent observations, which inflates the effective sample size and shrinks p-values towards nothing.
• Bad controls. Adding covariates because they are available. Conditioning on a mediator removes the effect you were trying to measure; conditioning on a collider creates an association that does not exist. Both look like being careful.
• Aggregation reversals. Simpson’s paradox is not exotic; it arises whenever group sizes differ and the grouping relates to the outcome. A model given a flat table has no way to know which stratification matters.
• Significance read as importance. A tiny effect in a large dataset is significant and may be irrelevant. Effect sizes and intervals get omitted unless requested.
• Selection built into the dataset. Survivors, responders, patients who reached a clinic. The model sees columns, not how the rows came to exist, and it cannot ask.
• Arithmetic and denominators. Rates computed against the wrong base, percentages of percentages, and the ordinary numeric slips described in why models are unreliable at arithmetic. Code execution helps here and does not eliminate it, because the wrong denominator computes perfectly.

The conversational interface is the trap

Here is the failure that does the most damage and gets discussed the least. You ask a question. You get an answer you did not expect. You say “try it another way”. You get another answer. You keep going until something looks right, and then you write it up.

That is p-hacking. It has always been possible, but it used to cost an hour of work per attempt, and the friction was doing real epistemological work. A chat interface reduces the cost to a sentence, so a dozen analyses can be run in the time it takes to think about whether the first one was appropriate. Nothing in the final write-up records that eleven others existed.

The defence is procedural rather than technical: decide the analysis before you look at the outcome, write it down, and count every departure from it. If you ran twelve analyses, the honest report says so, and the twelve are much less impressive than the one — which is the point.

A workflow that surfaces all three

Before

• Write the question and the analysis plan in advance, including which test, which covariates and what would count as a null result. Give the model that plan rather than the dataset and an invitation.
• Describe how the data came to exist — the sampling, the exclusions, the units, the sentinel values. Nearly every statistical error above is the model not knowing something you know and were never asked.
• Ask it to state the assumptions of the proposed test and how each would be checked, before running anything.

During

• Work in scripts, not in conversation. A script is reviewable, re-runnable and diffable; a chat transcript is none of those.
• Require row counts at every step, and the number of missing values per column at the start.
• Review the code, not the summary. The summary is the part the model is best at writing and the part that carries no information about correctness.

After

• Ask it to write the interpretation of the opposite result. If that paragraph is equally convincing, the original paragraph was not evidence-driven, and this is a fast, brutal check.
• Ask for the strongest argument against the conclusion, and for what confound would produce this pattern with no real effect.
• Record the model, version, prompts and every analysis run, per the reproducibility requirements.