Made agent eval runs survive code edits and refuse stale prompts

Situation

Agent evals (evals/eval run) ran inside the local dev agent server, which hot-reloads on every .py edit. Iterating on the agent while an eval ran (the normal loop) killed the eval partway through. A run of 5 to 75 simulated conversations was lost, along with its LLM spend and the wait. Worse, prompt and skill edits were not picked up at all, so a run could silently measure the old prompts and be read as a result for the new ones.

What I did

• Made evals/eval run start a throwaway agent server per run, from a snapshot of the working tree and without hot reload, so later edits and dev-server restarts cannot touch it. It exits when its run is done. Startup costs about 3 seconds.
• Made the dev server refuse a run, naming the changed files, when prompt or skill files changed on disk after it started. A review caught that my first version also flagged eval fixture files, which are read fresh each run; I narrowed it to what is actually cached.
• Made runs whose server died show as ABORTED with the reason, checked on every read, instead of showing as "still running".
• PR: https://github.com/Qwestly/qwestly-agent/pull/512 (merged to staging 2026-10-14). Ticket: https://app.asana.com/1/1208967535790507/project/1216663810328535/task/1219515353832377

Result

Pending. Expect aborted runs to drop to near zero for runs started on the isolated server.

Metrics

Metric	Baseline (date)	Target	Latest (date)
Local eval runs aborted by their server dying, of all runs	3 / 56 = 5.4% (2026-10-08 to 2026-10-14)	~0 on isolated runs	—

The baseline excludes 2 runs (03975130c0ba, 0cfcf8cc1e44) aborted on purpose while validating the change. The window is short because the local database starts on 2026-10-08. Each lost run was 1 to 75 attempts.

How to measure

Local MongoDB, candidate_portal.eval_runs, Dom's machine only (other developers' runs are not visible). Count runs by started_at window; an abort is error starting with abandoned or aborted (the new code also sets aborted: true). Split by owner.isolated once available:

mongosh --quiet "mongodb://localhost:27017/candidate_portal?authSource=admin" --eval '
printjson(db.eval_runs.aggregate([
  {$match: {started_at: {$gte: new Date("2026-10-15")}}},
  {$group: {_id: {isolated: "$owner.isolated",
                  aborted: {$regexMatch: {input: {$ifNull: ["$error", ""]}, regex: /^(abandoned|aborted)/}}},
            n: {$sum: 1}}}]).toArray())'

Caveat: a restore of the production database replaces local data, so take the measurement before any restore.

Log

• 2026-10-14: baseline taken (3 of 56 runs aborted); change merged to staging.