Extracting Text From Handwritten Meeting Notes

Recognising the letters is the part that is already solved well enough. What breaks on a page of meeting notes is everything the note-taker encoded in position, symbol and abbreviation — conventions they invented for themselves, in a hurry, and never wrote down.

What notes have that prose does not

A handwritten letter is linear text and the general handwriting problem covers it. Meeting notes are not linear. They are a spatial artifact produced under time pressure, and they carry at least five conventions that a straight transcription destroys.

• Position is meaning. A name in the left margin beside three lines owns those three lines. Read inline, it becomes a word in the middle of a sentence.
• Symbols are operators. An arrow means “therefore”, “assigned to”, or “see below”. A box drawn around a line marks it as an action. A star or double underline is emphasis. A vertical bracket spans the lines it applies to.
• Repetition is elided. Ditto marks and the abbreviation for them stand for the value on the line above, and a transcription that renders them literally loses the value entirely.
• Abbreviation is personal. Every regular note-taker has a private shorthand, and it overlaps only partly with anyone else’s.
• Uncertainty is marked. A trailing question mark on a figure means the writer was not sure of it. That is metadata about the value, and it must not be dropped or read as punctuation.

None of these is a recognition problem. They are all layout and convention, which is why the general handwriting-to-text page does not cover them and why a pipeline that only asks for a transcription will produce something fluent and structurally wrong.

The margin is a column of data

The margin convention is worth handling explicitly because it is nearly universal and because it is where the owners are. A page of notes typically has a wide body column and a narrow left or right margin containing initials, dates, stars and question marks against particular lines.

To keep it, you have to keep coordinates. Ask the model for regions with bounding boxes rather than for a single block of text, and segment the page into columns before transcribing so that margin content is transcribed as margin content. Then the association is geometric: a margin annotation applies to the body lines whose vertical extent it overlaps, and a bracket or brace applies to everything within its vertical span.

Two things to get right. The margin may be on either side and may switch sides between pages, so detect it from the layout rather than assuming. And a vertical line down the margin is either a bracket spanning lines or a printed rule on the paper — ruled and gridded notebooks are the norm — so distinguish printed rules by their regularity and their presence on unwritten parts of the page before treating one as an annotation.

Arrows deserve their own treatment because they connect two places on the page rather than sitting at one. Capture the arrow as a relation with a source and a target region, not as a character. An arrow from a line of text to a name in the margin is an assignment, and that is exactly the relation you wanted from the page.

Abbreviations are per-author

Some abbreviations are general and a model handles them without help. Many are local: a project code, a room name, a system nobody outside the team has heard of, and initials for people who attended. Those are where the errors concentrate, and they are also the highest-value tokens on the page, because they are the owners and the subjects.

Build a glossary per author and per organisation and pass it in the prompt as context. It is a small artifact and it compounds: after the first few pages of one person’s notes, most of their shorthand is in it, and every subsequent page is easier. Seed it from the organisation’s own vocabulary — project names, team names, system names — and grow it from the review queue, where a human correcting an expansion is the cheapest possible way to acquire a new entry.

Initials in particular should be resolved against the attendee list for that meeting rather than against a global directory, for the same reason the attendance block is the local namespace in formal minutes. Two people in a company can share initials; two people in one room usually do not.

Transcription and expansion are two fields

This is the rule that matters most and the one most easily broken by a prompt that sounds reasonable. Ask a capable model to “transcribe these notes” and it will produce prose. It expands the abbreviations, resolves the arrows into sentences, fixes the grammar and drops the question marks. Something like “PO 2wks late?? chase RB” comes back as “The purchase order is two weeks late and should be chased with RB.”

Read those two carefully. The original records uncertainty about the delay and does not assert that the purchase order is late. The output asserts it flatly, and it asserts it in fluent prose that a reader will trust more than the scrawl it came from. The model did not misread a single character; it did what it was asked and produced text, and in doing so it converted a note into a claim.

So require two fields and make the instruction explicit about both. A verbatim field containing the graphemes as written, abbreviations intact, punctuation intact, uncertainty markers intact, with a documented placeholder for anything illegible. And an expansion field which is allowed to interpret, carries the glossary version, and can be regenerated. Never one field, and never a verbatim field that quietly got tidied — the whole audit value of the transcription rests on it being the thing on the paper.

Ink colour and crossings-out are evidence

Two properties of the physical page carry information that binarisation throws away, and both are cheap to keep.

Ink colour distinguishes passes. Notes written during a meeting in one pen and annotated afterwards in another are two different acts at two different times, and the annotations are frequently the actions, decisions and corrections — the most valuable content on the page. Capture in colour and record a colour cluster per region rather than converting to bitonal at ingest. The same applies to highlighter, which is a deliberate emphasis marker and which turns into an indiscriminate grey smear the moment you threshold.

Crossings-out are data. A struck-through line is something the writer decided against, and in a meeting that usually means a figure was corrected or a decision reversed. Dropping it silently loses the reversal; transcribing it inline without marking it produces a page asserting both values. Mark the region as struck and keep the text, and where a replacement is written above or beside it, capture the relation between the two so the sequence is recoverable.

What to store

{
  "page": 1,
  "regions": [
    { "region_id": "m1", "kind": "margin_annotation",
      "bbox": [64, 420, 96, 40], "ink": "blue",
      "verbatim": "RB", "expansion": "R. Boateng",
      "expansion_source": "attendee_list", "confidence": "high" },
    { "region_id": "b7", "kind": "body_line",
      "bbox": [180, 418, 690, 44], "ink": "blue",
      "verbatim": "PO 2wks late?? chase",
      "expansion": "purchase order two weeks late (uncertain); chase",
      "glossary_version": 4, "struck": false,
      "uncertainty_marker": true },
    { "region_id": "b8", "kind": "body_line",
      "bbox": [180, 462, 690, 44], "ink": "red",
      "verbatim": "£4,200", "struck": true,
      "replaced_by": "b9", "note": "later pass" }
  ],
  "relations": [
    { "kind": "arrow", "from": "b7", "to": "m1" },
    { "kind": "brace", "from": "m2", "to": ["b10", "b11", "b12"] }
  ]
}

Regions plus relations rather than a text blob is what makes the margin, the arrows and the braces survive. Everything downstream reads the relations: an arrow from a body line to a margin initial is a candidate action with an owner, which is the same record formal minutes produce from much better-behaved input, and it should flow into the same schema with a much lower confidence and a route to review.