Posting Eval Results as a Pull Request Comment

Nobody opens CI logs. A reviewer looking at a prompt change wants to know which cases moved and by how much, and the only surface they are reliably looking at is the pull request page — so that is where the numbers have to be.

What the comment is for

Be clear that the comment is not the gate. The gate is the exit status from the runner, described in failing a build when the eval score drops below a threshold, and it fails independently of whether any comment was posted. The comment exists so that a human reading the diff can answer a question the red or green tick cannot: which of these cases changed, in which direction, and is that the change I intended to make.

That shapes the content. A single aggregate number is nearly useless in a comment, because a reviewer cannot act on it. What is worth the space is the per-metric table with the delta against the base branch, and then a short list of cases whose result flipped in either direction — flipped to failing because that is the regression, and flipped to passing because an unexpected improvement is often a case that is now being scored wrongly. Keep the whole thing under about twenty lines; a comment that needs scrolling gets collapsed and then ignored, which is the same fate as the log.

The permissions it needs

Workflow tokens are scoped per workflow, and the default scope in many organisations is read-only. Posting a comment needs write access to pull requests, which is granted with a permissions block. Grant it at the job rather than the workflow level so that the job doing the model calls is not also the job holding a write token, and keep contents at read.

permissions:
  contents: read
  pull-requests: write

If the comment step fails with a 403 and the token is correctly permitted here, the next thing to check is the organisation-level setting for default workflow permissions, which can cap what a workflow is allowed to request no matter what the file asks for. Repository settings cannot escalate past the organisation policy.

Declaring a permissions block at all switches the token from whatever the default is to exactly what you listed. That is the behaviour you want — it is explicit, and it means an eval job with no business writing to the repository cannot — but it does mean adding the block to fix the comment step can break an unrelated step in the same job that was relying on a broader default. If something else stops working the moment you add this, that is the cause, and the fix is to name the additional scope rather than to remove the block.

The workflow

name: eval

on:
  pull_request:
    branches: [main]

jobs:
  eval:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write
    steps:
      - uses: actions/checkout@v4

      - name: Score the golden set
        id: eval
        env:
          EVAL_API_KEY: ${{ secrets.EVAL_API_KEY }}
        run: |
          python -m evals.run \
            --cases evals/cases.jsonl \
            --thresholds evals/thresholds.json \
            --json-out reports/eval.json \
            --markdown-out reports/eval.md
        continue-on-error: true

      - name: Comment the results
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          gh pr comment "${{ github.event.pull_request.number }}" \
            --body-file reports/eval.md \
            --edit-last --create-if-none

      - name: Fail if the gate failed
        if: steps.eval.outcome == 'failure'
        run: exit 1

The three-step shape is the load-bearing part. continue-on-error: true on the eval step lets the comment step run even when the score failed — which is precisely the run where the reviewer needs the table — and the final step restores the failure so the check is still red. Skip that last step and you have built a very informative gate that does not gate.

One comment, edited

A new comment on every push turns a pull request with eleven commits into a thread of eleven near-identical tables, and the notification traffic makes people mute the pull request. The GitHub CLI handles this directly: --edit-last edits the last comment by the current user, and --create-if-none, which can only be used together with it, creates one when there is nothing to edit. That pair gives you a single comment that updates in place, with no state to store and no marker comment to search for.

One consequence worth knowing: “the current user” is the identity of the token. If some runs use the workflow token and others use a bot app installation token, they are different users and you get two sticky comments rather than one. Pick one identity for the comment and keep it.

Fork pull requests cannot post

Here is the wall. For a pull request opened from a fork, the workflow token is read-only regardless of the permissions block, and repository secrets are not exposed to it. So both halves of this workflow fail on exactly the contributions from outside your organisation: no key to call the model with, and no write access to post the result.

That restriction is deliberate, and the way round it that appears first in search results — switching the trigger to pull_request_target — is the dangerous one, because that trigger runs with a writable token and access to secrets in the context of the base repository. If such a workflow checks out and executes the pull request’s code, you have handed an arbitrary contributor your provider key. For an eval workflow, whose whole job is to execute the prompt files from the branch under review, that is not a theoretical concern.

• Split the run from the post. Score in a workflow triggered by the pull request, upload the markdown as an artifact, and post it from a second workflow triggered on the completion of the first, which runs in the base repository context with a writable token and never executes fork code.
• Or require approval. Configure the repository so workflows on fork pull requests need a maintainer to approve each run. This is a human gate on spending your key, which for an eval suite is the right default anyway.
• Never check out the pull request head under a trigger that has secrets. If you take one rule from this section, that is it.