AI in Legal: Contract Review and Its Limits

A contract review tool is not reading a contract. It is answering a question across a family of documents that amend each other, in which the thing you are looking for may be defined in one, modified in another, and waived in a side letter nobody scanned. This page is not legal advice, and it makes no claim about what any bar or regulator requires.

What a contract actually looks like

Demonstrations use a clean, born-digital PDF of a single agreement. Production does not. An executed contract arrives as a scan of a printed copy, often with a signature page photographed separately and appended, occasionally upside down. The governing set is a master agreement plus statements of work, order forms, amendments and side letters — and an amendment does not restate the clause it changes. It says that section 8.3 is deleted and replaced, or that the third sentence is struck. So the question “what is the notice period for termination for convenience” cannot be answered from any one document in the pile.

This shapes the pipeline before any prompt is written. The unit of retrieval is the matter or the counterparty, not the file. Documents need an execution date and a relationship to a parent, which is often only recoverable from the text itself (“this First Amendment to the Master Services Agreement dated 4 March”). Getting that graph right is an extraction task in its own right, and it is the one that decides whether everything downstream is answering the current question or a superseded one.

Recall is the number, not accuracy

Legal review tasks are lopsided in a way that makes a single accuracy figure meaningless. In a due diligence pass, the job is to find every contract containing a change-of-control assignment restriction. A false positive costs a lawyer perhaps thirty seconds of reading. A false negative costs a deal term that nobody priced.

So the evaluation is per-clause-type recall at a stated precision, on a set of documents a lawyer has already marked up, and it is reported per clause type rather than averaged. Limitation of liability is formulaic and easy; the same system will find it in almost every document and miss an exclusivity commitment buried in a schedule. Averaging those into one number hides exactly the clause type that will cause the problem. If you have never built one of these, the general method is in building a golden dataset and the statistics of “is this actually better” in evaluation statistics.

One design choice buys more recall than any prompt change: let the system return not_found and mean it. A model asked “what is the cap on liability” will produce a number, because a number is the likely continuation. A schema with an explicit absent case, and an evaluation that scores absent-when-absent as correct, converts a confident wrong answer into a routed question. See extraction confidence for how to make that signal usable rather than decorative.

The defined-term trap

Contracts are written in a private language declared at the top. “Affiliate”, “Confidential Information”, “Services” and “Losses” are capitalised because they mean whatever section 1 says they mean, and the definition is frequently by reference to another agreement. Two documents in the same pile can define Affiliate differently — one requiring majority ownership, the other any common control — and a model that answers from the wrong one produces a sentence that is fluent, plausible and wrong in a way that is very hard to spot on review, because the error is not in the clause it quoted.

Mitigations are unglamorous and they work: extract the definitions section first and carry it in context with the clause under examination; refuse to answer a question about a defined term when the definition was not retrieved; and flag when the retrieved definition came from a different document than the retrieved clause. That last check is a join, not a model call.

What the signing lawyer needs

A partner or in-house counsel puts their name on the advice. The accountability does not move, which means the output has to be verifiable in less time than doing the work would have taken — otherwise the tool is a cost. That is a concrete interface requirement, not a philosophy:

• A span, not a summary. Every extracted fact carries the document id, the page, and the character offsets of the text it came from, so the reviewer lands on the sentence rather than searching for it.
• The quoted text verbatim. Checking that a quoted string appears exactly in the source is a string comparison you run before display. It is the cheapest hallucination control available and it eliminates the invented-clause failure entirely.
• The unanswered list. A review of forty contracts that returns thirty-eight answers and two flagged gaps is useful. One that returns forty answers is not trustworthy, and a reviewer cannot tell which two to check.

Confidentiality is the other constraint that shapes the architecture. Client documents are held under duties of confidence, and engagement terms and outside counsel guidelines often speak to where those documents may be processed and by whom. Whatever the specific duty in a given jurisdiction and matter, the engineering consequence is predictable: data residency, retention and sub-processor questions get asked before the pilot, not after. The pages on zero data retention and data processing agreements cover what to ask for.

Where the work is not the document

Two adjacent tasks are much better fits than review and get less attention. The first is playbook triage: on inbound third-party paper, comparing each clause to the organisation’s own standard positions and fallback positions, and sorting into accept, negotiate, escalate. The comparison target is a document the organisation controls, the taxonomy is small, and a wrong sort costs a human glance.

The second is obligation extraction into a calendar: renewal windows, notice deadlines, audit rights, insurance certificates due. These are structured facts with dates, they are checkable, and missing one is a pure operational loss that no lawyer enjoys owning. Both have the property that makes an early legal project survivable — the failure is visible immediately and costs minutes, rather than surfacing in eighteen months as a term nobody knew they had agreed to.