Shipped the role fit assessment on hire matching's own scorer, with eval baselines

Situation

Members bring jobs they found elsewhere and ask their agent "am I a fit?". Phase 1 could save a role but forbade any verdict, because the agent had nothing better than a guess. The plan required the verdict to come from the same scorer a hiring manager's match run uses, never to show a score, and never to suggest claiming a skill the member does not have.

What I did

Built assess_role_fit (qwestly-agent #444, candidate #1172): "their bar" calls hire matching's own filter, deterministic components and LLM scorer without changing a line of lib/hiring_matching/; "your bar" checks work mode, location, level, pay deterministically; member notes override the posting ("recruiter said remote is fine"); a gap sort files showcase / real / hard-mismatch gaps and writes a versioned fit report.
Enforced the honesty rules in code, not just the prompt: a showcase quote must appear in the member's own material or it becomes a real gap; a hard mismatch never gets work-towards advice and forces "not a fit".
Found and handled a new Claude Sonnet 5 output failure (list fields sent as JSON text, repeated on every retry) that failed 13 to 21 of 135 assessments; 0 after.
Built two evals and recorded first baselines; ran the plan's 10 live scenarios.

Result

Baselines, 2026-10-07 (qwestly-agent/evals/role_fit/baselines/):

Gap kinds, 15 synthetic cases: 0 unsupported showcase claims (the bar), 16/16 kinds correct, 0 hard mismatches missed.
Verdict agreement vs admin labels, 135 cohort members (index rows only, a lower bound): poor read negative 104/108; good read positive 3/5; acceptable positive 0/22; agreement 0.793, balanced 0.537.
Cost: ~$0.03 per assessment (synthetic) to ~$0.044 (cohorts), 10 to 22 s.

How to re-measure

From a qwestly-agent checkout: ../scripts/shared-env.sh --for . -- venv/bin/python -m evals.role_fit.runner gaps and ... runner verdicts --concurrency 8. Compare with the baseline JSON files. Once in production: count of role-fit documents and assessments per week (log line role fit assessed).

Next check

2026-10-21: bar for verdict agreement set; Phase 3 merged; first real-member usage counts if the flag is on.