Moving the free LinkedIn writer onto the agent skill system, against a measured baseline

Situation

The free LinkedIn writer is the first thing visitors from our outreach post use, and it still ran the original scripted state machine. It recorded any reply as an answer and moved on, so it could not push back on thin answers, act on a "yes", or notice an answer to the wrong question. Two outside testers (Oct 10-11) filed about 20 tickets against it. The signed-in linkedin_writer skill already handled this well, but pre-auth had never been ported.

What I did

• Wrote the migration plan (qwestly-workspace/plan/preauth-skills-surface.md): a Surface abstraction so any skill can run on pre-auth, per-conversation engine pinning for rollback, conversion rules, a session cost cap, and pre-auth upload carry-over at sign-up. Got product decisions resolved the same day.
• Built a before/after regression harness (qwestly-agent PR #500) that runs one suite against both engines: scripted user moments, engine-neutral checks for each tester ticket, quote-backed LLM judge questions, and per-session cost and latency.
• Established the flow-engine baseline with production settings (inferred the hidden production flag from PostHog generation history), 15 personas x 5 runs.

Result

Pending: the skill engine lands in Phases 2-3. Baseline already found that a closed ticket (104, title and date lines in rewritten entries) still occurs in 9 of 59 sessions.

Metrics

Metric	Baseline (2026-10-13, run fb70834f91e8, n=72 sessions)	Target	Latest
Main-model cost per session (excludes sub-LLM calls, so low)	mean $0.47, median $0.40, max $1.86	no worse after adding sub-calls	skill engine mean $0.73, max $2.38, all calls (2026-10-14, 70ab2349deab)
Slowest save turn	median 34 s, p95 54 s, max 81 s	p95 <= 54 s	median 16 s, p95 40 s (2026-10-14)
First save says the rewrite stays in Qwestly (R10)	0 of 70	>= 4 of 5 per persona	73 of 74 (2026-10-14)
Entries starting with a title/date line (R14, gating)	9 of 59 sessions	0	0 of 73 (2026-10-14)
Read names what a strong profile has (R7, judge)	fail 16 of 71	clearly fewer	-
Pushback on thin answers (R17, judge)	fail 15 of 61	clearly fewer	-
Real model spend per session (cache-aware, from PostHog)	flow mean $0.56, p90 $0.78 (2026-10-14)	lower	skill mean $0.21, p90 $0.28 (2026-10-14)
Answer to the wrong question re-asked (R2, judge)	fail 2 of 2	pass >= 4 of 5	fail 2 of 3 (2026-10-14)

How to measure

In qwestly-agent, with the local agent .env holding PREAUTH_GAP_BEATS_INLINE=true (production's value) and a running local candidate app: scripts/shared-env.sh --for qwestly-agent -- bash -c 'cd qwestly-agent && evals/eval run -a writer -k 5', then evals/eval report <run_id>. From Phase 2, compare the engines on the same build. Cost reads only the main model until the harness sums sub-calls.

Log

• 2026-10-15: shipped to staging (qwestly-agent #520, candidate #1216, public-site #230), engine on by default; final suite 1/75 sessions failed on the new engine vs 17/75 on the old flow, real cost $0.29 vs $0.55 per session, slowest save p95 36 s vs 52 s. Resolved 18 pre-auth tickets with regression evidence. Production rollout at the next release; re-measure one week after.
• 2026-10-14: fair cost comparison from PostHog (cache-aware, harness calls excluded, 74 sessions each): real spend per session $0.21 on the skill engine vs $0.56 on the flow engine (about 2.6x cheaper); p90 $0.28 vs $0.78.
• 2026-10-14: skill engine on the feature branch, full suite 15x5: 6 of 75 attempts failed vs 18 of 75 on the flow engine; cache share 91% vs 24%; fixed a shared edit_document loop (no-break spaces in imported LinkedIn text) that could burn $4.88 in a turn.
• 2026-10-13: plan written, product decisions resolved, harness PR #500 opened, flow baseline recorded.