_private/qwestly-private-docs/SOC2/incidents/2026-05-21-openai-key-compromise.md

Table of Contents

Incident Report — 2026-05-21

OpenAI API Key Compromise

Date: 2026-05-21 Reported by: Dominick Pham Severity: P1 — High (production services affected) Classification: Security Incident Closed: 2026-05-21

Summary

OpenAI notified us via email that one or more API keys associated with our account had been leaked/compromised. OpenAI did not disclose how the keys were compromised. They disabled the affected keys, causing all services relying on them to break. Coincidentally, Vapi (our voice API provider) experienced a simultaneous but unrelated platform outage, which led to initial confusion about whether the two events were connected.

Impact

  • All services relying on the compromised OpenAI keys stopped functioning
  • Production apps and staging environments were impacted
  • Vapi voice chat was also down during this window due to an unrelated Vapi platform outage (see Appendix A)

Timeline

Time (EDT) Event
10:57 Received email from OpenAI: API keys compromised & disabled
10:57–12:20 Created new OpenAI API keys (separate keys for testing and production); redeployed all production apps & staging environments
12:20 All Qwestly services back online with new keys
~12:35 Noticed Vapi dashboard was also down (unrelated CORS error)
12:45 Checked Vapi status page; confirms dashboard and API outage
~13:15 Vapi status page reports dashboard service restored
~13:50 Vapi status page reports API service restored
~14:00 Confirmed interview feature on Qwestly fully functional

Resolution

  • Created new OpenAI API keys, with separate keys scoped to testing and production environments
  • Redeployed all production applications and staging environments with the new keys
  • Verified all services restored by 12:20 EDT (approximately 1 hour 23 minutes from detection)

Root Cause Analysis

OpenAI did not disclose how the key was leaked. The key was not stored in Vapi (Vapi bills us for OpenAI model usage but does not receive or store our API keys). The simultaneous Vapi outage was coincidental — Vapi published an RCA confirming their outage was caused by a database provider misconfiguration that exhausted Postgres connection pools, not a security incident (see Appendix A).

The most likely leak vectors, in order of probability:

  1. Developer workflow exposure — key copied in chat messages, terminal history, or local .env files inadvertently committed or shared
  2. Third-party service with key access — a service integrated with our OpenAI key (e.g., LangSmith) may have experienced its own compromise
  3. OpenAI-side detection — OpenAI may have flagged the key based on heuristic detection of anomalous usage patterns, not an actual external leak

Without OpenAI disclosing the detection method, a definitive root cause cannot be established. However, the key rotation and environment separation address the immediate risk regardless of leak vector.

Lessons Learned

  1. Key scoping matters. The compromised key was shared across production and staging. Creating separate keys per environment limits blast radius.
  2. 1:23 is a reasonable MTTR for a key rotation. The redeployment process worked smoothly. No automation bottlenecks were encountered.
  3. Correlation is not causation. The simultaneous Vapi outage was coincidental. Avoid assuming two incidents are linked without evidence.
  4. OpenAI's notification was clear and actionable. Their automated compromise detection provided sufficient detail to act immediately.

Follow-up Actions

  • Rotate keys for dev services using the compromised key (e.g., LangSmith) — completed 2026-05-21
  • Notify team members using dev keys to update — completed 2026-05-21
  • Review key management practices — evaluate secrets manager or vault for centralized key storage
  • Implement routine key rotation schedule (quarterly minimum)

Appendix A: Vapi Platform Outage (Coincidental)

Vapi experienced a simultaneous but unrelated platform outage on 2026-05-21. Per Vapi's published RCA:

  • Root cause: A database provider applied a misconfigured audit logging change at 6:44 AM PT that caused Postgres processes to block while writing log output, exhausting connection pools on two critical databases
  • Duration: Approximately 4 hours 7 minutes (7:02 AM – 11:09 AM PT)
  • Impact: Voice calls failed, API returned 5xx errors, dashboard was inaccessible
  • Resolution: Provider disabled the affected audit logging and restarted the database; Vapi ramped traffic back gradually
  • Note: Vapi does not store our OpenAI API keys. They bill us for OpenAI model usage through their platform.

Vapi published their full RCA at the time of the incident. This appendix documents the event for completeness and to clarify that it was unrelated to the OpenAI key compromise.