Timing Out a Stuck Eval Job Without Losing the Whole Pipeline

The job did not fail. It sat at the same log line for six hours and was then cancelled by the platform, having produced no test report, no artefacts and no indication of which case it was on.

The symptom

The log ends with a bare cancellation:

Error: The operation was canceled.

and the run page carries an annotation to the effect that the job exceeded the maximum execution time of 360 minutes. That 360 is not a coincidence and not a quota you have exhausted: it is the default value of jobs.<job_id>.timeout-minutes, applied because you never set one. GitHub’s documentation states the default plainly, and the field is the maximum number of minutes to let a job run before it is automatically cancelled.

Three things make this worse than a normal failure. The runner was held for six hours, which on a constrained allocation blocks everything behind it. No artefacts were uploaded, because a cancelled job skips its remaining steps. And you cannot tell from the log which case hung, because the suite’s reporter had not written anything yet.

Set the job timeout first, before diagnosing anything. It converts a six-hour outage into a twenty-minute failure and costs one line:

jobs:
  eval:
    runs-on: ubuntu-latest
    timeout-minutes: 20

That is containment, not a fix. The rest of this page is the fix.

Why the request never returned

There are five causes and they are distinguishable, which matters because they need different timeouts.

• A stalled stream, which is the interesting one. A read timeout fires when no bytes arrive for N seconds. A streaming response that has stopped producing content but is still emitting keep-alive traffic — an SSE comment line, a periodic ping event — is delivering bytes, so the read timeout resets every time and never fires. The connection is alive, the stream is not progressing, and nothing in the client’s timeout model is watching total elapsed time. This can and does hang until the job dies.
• The default timeout is much longer than you assume. The OpenAI Python SDK’s README states that requests time out after ten minutes by default, and that certain errors are automatically retried twice with a short exponential backoff. Those compose: one case can consume roughly thirty minutes before raising, and a suite of two hundred cases has plenty of room to reach six hours without a single infinite hang.
• Retry loops without a global deadline. A per-attempt timeout bounds one attempt. Wrap it in your own retry loop on top of the SDK’s and the bound multiplies again.
• Connection-pool exhaustion. If concurrency exceeds the client’s pool size, requests queue waiting for a connection, and time spent waiting for a pool slot is usually not covered by the read timeout. The suite appears to hang with no request in flight.
• A rate limit being absorbed. Automatic retries of 429 responses with backoff turn a throttle into a slow suite rather than an error. Multiply by a shard count and this alone can be hours.

Telling them apart afterwards is possible if the job left anything behind, which is the argument for the step-level timeout and the always-upload in the last section. Three cheap signals separate them. If the last log line is a case starting and no case ever completed after it, you have a hang on a specific case — a stalled stream or a pool wait. If cases kept completing but at a steadily falling rate, you have absorbed rate limiting. And if the process produced no output at all after the install step, suspect the pool: a client whose connection limit is below the suite’s concurrency will accept every task and start almost none of them.

Emitting a heartbeat makes this diagnosis trivial rather than inferential. A line every ten seconds naming the number of in-flight requests and the oldest one’s age costs nothing and turns “the job hung” into “case 87 has been in flight for nine minutes”. Unbuffer the output when you do it, or the heartbeat sits in a pipe buffer and is lost with the cancellation.

Timeout one: the HTTP client

Set an explicit timeout on the client and stop relying on the default. For the OpenAI Python SDK this is a constructor argument, overridable per request:

from openai import OpenAI
import httpx

client = OpenAI(
    # Granular is better than one number: a connect timeout of 2s fails fast
    # on a dead route, while the read timeout governs inter-chunk gaps.
    timeout=httpx.Timeout(60.0, read=10.0, write=10.0, connect=2.0),
    max_retries=2,   # the default; set 0 if your harness retries
)

# Per-request override, for one case you know is slow.
resp = client.with_options(timeout=120.0).chat.completions.create(
    model="gpt-5.5",
    messages=[{"role": "user", "content": prompt}],
)

The two numbers to think about are the connect timeout, which should be short because a slow connect is never going to become a fast request, and the read timeout, which under streaming is a gap budget rather than a total. Set the read timeout to a little more than the longest plausible pause between chunks, not to how long you are willing to wait overall.

Then decide who owns retries. If your harness has its own bounded retry policy, set max_retries=0 and own it fully. Two independent retry layers multiply, and neither knows the other exists.

Timeout two: a wall-clock deadline

Because the read timeout cannot see a keep-alive stall, a total deadline around the whole call — including consumption of the stream — is the timeout that actually catches the case in the first bullet above. It is a few lines and it is the one people are missing.

import asyncio

async def run_case(client, case, *, deadline_s: float = 90.0):
    async def _call():
        chunks = []
        stream = await client.chat.completions.create(
            model=case.model,
            messages=case.messages,
            stream=True,
        )
        async for event in stream:
            delta = event.choices[0].delta.content
            if delta:
                chunks.append(delta)
        return "".join(chunks)

    try:
        return await asyncio.wait_for(_call(), timeout=deadline_s)
    except asyncio.TimeoutError:
        raise RuntimeError(
            f"case {case.id}: no completion within {deadline_s}s "
            f"(stream stalled or provider slow)"
        ) from None

Two properties make this worth the code. The error names the case, so the log tells you which one hung — the thing the six-hour cancellation could not. And the failure is a distinct class, so it can be excluded from the retry policy that covers transport errors: a stalled stream retried is a stalled stream repeated.

Timeouts three and four: test and job

Above the deadline sit two more, each catching what the one below it missed.

1. Per-test. Vitest’s testTimeout and pytest’s timeout plugin bound one case even if your own deadline logic has a hole in it. Set it a little above the wall-clock deadline so that in the normal case your error message wins, and this layer only fires when the deadline itself failed to.
2. Per-step. timeout-minutes is available on individual steps as well as jobs. Putting it on the eval step rather than only the job is what lets subsequent steps still run — the step fails, and an upload step marked if: always() still executes. This is the difference between a timeout you can diagnose and one you cannot.
3. Per-job. Twenty minutes, or whatever a healthy run plus a wide margin is. The purpose of this one is to bound the damage from anything the other three did not anticipate, including the runner itself getting stuck.

jobs:
  eval:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v7
      - run: npm ci
      - name: Run evals
        timeout-minutes: 12          # step fails; later steps still run
        run: npx vitest run eval/ --reporter=junit --outputFile=junit.xml
      - uses: actions/upload-artifact@v7
        if: always()                 # without this, a timeout uploads nothing
        with:
          name: eval-report
          path: junit.xml
          retention-days: 7

One more line prevents the pipeline-wide version of the problem. Stuck jobs stack up when pushes arrive faster than runs complete, and a concurrency group cancels the superseded ones:

concurrency:
  group: evals-${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

Finally, if provider hangs are frequent rather than occasional, the right structure is not a longer timeout but a circuit breaker: after a small number of deadline failures, stop calling and fail the run immediately with a clear message. A suite that spends fifteen minutes discovering the provider is down should discover it in thirty seconds instead, and the retry policy should be the thing that decides whether that is worth another attempt.