Skip to content
Multigrid

Storing Eval History Across CI Runs to Chart a Trend

11 min read · updated August 11, 2026

A single eval score answers “did this pass”. A sequence of them answers “is this getting worse”, which is the more useful question and the one nobody can answer from CI logs that expire. The mechanism is small. The metadata is what makes it work.

What a row has to contain

The score alone is almost useless. Three months from now the line will drop four points and somebody will have to work out why, from a repo where the prompt, the dataset and the model have all moved. Every field below exists to make one specific explanation checkable.

  • Commit SHA and timestamp. The SHA is the join key to everything else; the timestamp lets you plot against wall time, which is how a provider-side change presents itself.
  • Dataset fingerprint. A hash of the case file, plus the case count. Scores computed over different datasets are not comparable and must not be drawn on the same line — this is the single most common way a trend chart lies.
  • Model identifier, as specific as the provider will give you. Not the alias you requested but whatever version string came back in the response. An alias that silently repoints to a new snapshot is precisely the change a trend is supposed to surface, and it is invisible if you only record what you asked for. This is the mechanism behind silent model updates.
  • Sampling settings. Temperature, top-p, and any seed. A drop that coincides with somebody raising temperature for an unrelated feature is otherwise an unsolvable mystery.
  • Prompt version. A hash of the assembled prompt, not of the template file, so a change in an included partial is reflected.
  • Per-metric scores and the denominator. Store passed and total rather than a rounded rate. You can always derive the rate; you cannot recover the counts, and the counts are what tell you whether a two-point move is meaningful.
  • Retry count and total tokens. Cheap to record and they turn the same file into a cost and reliability trend, which is the other question people ask of it.
{"ts":"2026-08-11T09:14:22Z","sha":"9f2c1ab","branch":"main",
 "dataset":"sha256:41d0e9…","cases":200,
 "model_requested":"gpt-5.5","model_returned":"gpt-5.5-2026-06-18",
 "temperature":0,"seed":7,"prompt_hash":"sha256:b1c9…",
 "metrics":{"schema_valid":{"passed":200,"total":200},
            "tool_correct":{"passed":191,"total":200},
            "judged_quality":{"passed":178,"total":200}},
 "retries":3,"tokens_in":248310,"tokens_out":61240}

Where to put it

The obvious storage options are each wrong in an instructive way.

Build artefacts are per-run by construction and expire; retention is finite and configurable, and a trend you cannot read back beyond the retention window is not a trend. The Actions cache is worse for this purpose, not better: it is explicitly an evictable cache, entries are removed when unused and the store has a size limit per repository, so the correct mental model is that any entry may vanish between runs. A comment on the pull request is a fine presentation layer and a terrible store, because it disappears when the branch is deleted.

What you want is something append-only, cheap, and living as long as the repository. An orphan branch holding a single JSONL file does the job with no infrastructure at all: it is versioned, it is diffable, it is not in anyone’s working tree, and it does not pollute the main branch’s history with a commit per CI run.

# Once, locally.
git checkout --orphan eval-history
git rm -rf .
printf '' > history.jsonl
git add history.jsonl
git commit -m "start eval history"
git push -u origin eval-history
git checkout main

If your volume outgrows a text file — thousands of runs, or many metrics per run — move to a database or an object store and keep the same row shape. Nothing above depends on the storage choice.

Appending without losing a run

The failure this step has is concurrency. Two workflow runs finishing within seconds of each other both fetch the branch, both append, and the second push is rejected as non-fast-forward. Handled naively, one run’s row is silently lost; handled badly, the job fails and people start ignoring it.

Because the file is append-only and rows are independent, the retry is trivially correct: fetch again, re-append, push again. A short bounded loop is all it takes.

  history:
    needs: eval
    if: always() && github.event_name == 'push'
    runs-on: ubuntu-latest
    permissions:
      contents: write
    steps:
      - uses: actions/checkout@v7
      - uses: actions/download-artifact@v7
        with: { name: eval-summary, path: . }
      - name: Append to eval-history
        run: |
          git config user.name  "github-actions[bot]"
          git config user.email "github-actions[bot]@users.noreply.github.com"
          for attempt in 1 2 3 4 5; do
            git fetch origin eval-history
            git worktree add --force ../hist origin/eval-history
            cat summary.json | tr -d '\n' >> ../hist/history.jsonl
            echo "" >> ../hist/history.jsonl
            git -C ../hist add history.jsonl
            git -C ../hist commit -m "eval: ${{ github.sha }}"
            if git -C ../hist push origin HEAD:eval-history; then
              exit 0
            fi
            git worktree remove --force ../hist
            sleep $(( attempt * 3 ))
          done
          echo "could not append eval history after 5 attempts" >&2
          exit 1

Three deliberate choices in that block. The job needs permissions: contents: write explicitly, because the default token permissions in many organisations are read-only and the failure is a 403 at push time, well after the expensive work. if: always() means a failing eval run still records its score, which is the run you most want on the chart. And it is restricted to push events so that pull-request runs — which are transient and often re-run — do not fill the series with noise from branches that never landed.

Turning it into something you look at

A history nobody sees is a log file. Two cheap presentation layers cover most of the value.

  1. A run summary on every eval job. Write a Markdown table to $GITHUB_STEP_SUMMARY comparing this run’s metrics with the last recorded row on main. The delta is what people actually read; the absolute number rarely is.
  2. A committed chart. A scheduled job reads the JSONL, filters to rows sharing the current dataset fingerprint, and writes an SVG back to the same orphan branch. An SVG in a repository renders in the browser and needs no dashboard, no service and no credentials.
  3. A flat-line alarm. Plotting is for humans; the useful automated read is a change-point check. A metric that has not moved in fifty runs is as suspicious as one that dropped — it usually means the eval stopped actually calling the model. That is the subject of blocking a deploy on a flat eval score.

What invalidates a series

Some changes make old and new points incomparable, and the right response is to start a new series rather than to explain the discontinuity in a comment nobody will find.

  • The dataset changed. Adding twenty hard cases will drop the rate even though nothing regressed. This is why the fingerprint is a field and why the chart filters on it: the line simply ends and a new one begins.
  • The scoring changed. If a judged metric is produced by a model, that judge is itself a version with its own drift. Record the judge model and prompt hash exactly as you record the system under test, or the trend is measuring two moving things at once.
  • The sampling settings changed. A temperature change moves the distribution, not just the mean.

Rows in a JSONL file are cheap and permanent, which is the property that makes all of this work: you can add a field next month and old rows simply lack it, without a migration and without invalidating anything already written.

Related