Established the org-wide vulnerability remediation baseline for SOC 2 evidence

Situation

The SOC 2 auditor's evidence request "records of security issues being tracked" had only ever been answered with one page: a single Asana issue from October 2025 and a screenshot. Nothing showed that security issues across the 2026 audit period (2026-07-02 to 2026-10-02) were actually tracked and closed, and no one had ever measured whether our own remediation promise was being met. The 2025-style submission also risked a follow-up request, since the auditor is mid-review with report issuance targeted for 2026-11-16.

What I did

I treated the request as a measurement problem before an evidence problem. I pulled the full Dependabot alert population for every in-scope repository and computed detection-to-fix intervals per severity, then reconciled the numbers against the merged pull request history so the fix path was provable, not asserted. I found the non-dependency half of the record too: I searched Asana (sprint boards and bug backlogs, where security work was actually living) for security issues rather than trusting labels, recovered the linked fix PR for each one, and applied the security tag retrospectively so the population is reviewable next period. I chose to disclose the retrospective tagging in the evidence itself, since the tag timestamp is visible and an unexplained one is worse than a stated one. I also excluded the internal auth-gated app from scope deliberately and named no internal-only repository anywhere in the auditor-facing record.

Result

Pending until the auditor's review concludes. The baseline is that 233 Dependabot alerts were opened and closed in the period, all 11 critical ones within a day, with 90.6% inside the 3-day on-call SLO; 214 automated fix PRs were merged; and 12 tracked Asana security tasks were resolved. The exercise surfaced the real gap the SLO had been hiding: 22 alerts ran past 3 days, including three high-severity ones at 48 and 63 days on repos that had no Dependabot configuration. That finding is what produced a Dependabot config for qwestly-agent and the retarget-to-staging workflow, so the number to watch next period is the over-SLO rate, not the alert count.

Metrics

Metric	Baseline (2026-07-02 to 2026-10-02)	Target	Latest
Dependabot alerts opened (in-scope repos, excl. internal)	233	Tracked, none silently dropped	233 (all closed)
Critical alerts fixed within 3 days	11/11 (100%)	100%	11/11
High alerts fixed within 3 days	91/99 (91.9%)	100%	91/99
All alerts within 3-day on-call SLO	211/233 (90.6%)	Raise to 100%	211/233
Alerts over SLO	22/233 (9.4%)	0	22/233
Worst high-severity time to fix (days)	63 (qwestly-agent, pydantic-ai) and 48 (qwestly-ui, sharp)	<= 3	63
Dependabot-authored PRs reviewed and merged	214	n/a	214
Asana security-tagged tasks resolved in period	12/12	100%	12/12

How to measure

Alert population, per repository, for the window:

for r in candidate qwestly-hire public-site api-python qwestly-agent candidate-catalog qwestly-ui docs qwestly-status qwestly-contracts; do
  gh api "repos/Qwestly/$r/dependabot/alerts?per_page=100" --paginate \
    --jq '.[] | ["'$r'", .number, .state, .security_advisory.severity, .security_vulnerability.package.name, .created_at[:10], (.fixed_at // .dismissed_at // "")[:10]] | @tsv'
done

Caveats: created_at is when GitHub recorded the alert, which for a repo whose Dependabot was enabled late can postdate the vulnerable dependency; time-to-fix is therefore an upper bound on exposure, and the three long-running alerts are the clearest example. A repo's alerts also only close once the fix reaches the default branch, so staging-branch fixes show as open until release. Asana side: GET /workspaces/<gid>/tags, then the tag's tasks; the security tag is 1211816302234381.

Log

• 2026-10-05: Alert and PR history pulled and reconciled; Asana security population assembled and tagged; the qwestly-agent Dependabot gap and the policy SLA conflict identified; evidence package drafted (narrative PDF plus five screenshots) for SOC 2 document "Records of security issues being tracked", due 2026-11-03.