Table of Contents

Dictation System Overview (updated 7/24/2026)

Voice-to-text dictation for the chat composer. The user taps a mic button, speaks, and the transcript is inserted into the chat input when they stop. Available in two surfaces: the onboarding agent chat (candidate app, authenticated) and the public agent chat (public-site, anonymous).

How it works (non-technical)
Architecture
Key modules
The recording lifecycle
Two finalization paths
API endpoints
Rate limits and safety
Error handling
Environments and limits
Design decisions
Glossary

How it works (non-technical)

The system works like a voice memo app built into the chat:

Tap the mic — recording starts immediately on your device. The UI shows an audio waveform so you can see that sound is being captured.
Speak — there is no visible timer during recording, just the waveform. You can talk for up to 10 minutes.
Tap confirm (✓) — the recording stops, uploads to our server, and is transcribed. The text appears in the chat input, preserving anything you already typed.
Or cancel (✕) — the recording is discarded without transcribing anything.

If transcription fails (server error, timeout), the recording is kept in memory. A "Retry" button appears so you can re-upload the same recording without re-speaking.

Architecture

Capture-first design

The core principle: recording is local, transcription is remote. The browser's MediaRecorder API captures audio directly on the device. Nothing depends on a server connection being ready — recording starts immediately, so no words are ever lost.

When the user stops recording, the complete audio blob is uploaded to a server endpoint which forwards it to OpenAI for transcription. This is the batch path — always available, always correct.

Opportunistic Realtime fast-path

There is an optional speed optimization, enabled on both surfaces (the shared ChatInputBar sets realtime: true for every consumer). On the first mic click a background WebRTC connection to OpenAI's Realtime API begins connecting. On subsequent recordings, if this connection is healthy, transcription finalization is faster because audio has been streaming as the user speaks. If anything goes wrong with the Realtime path — connection not ready, transport error, timeout — the system silently falls back to the batch upload. The user never notices a difference except that later recordings finish transcribing sooner.

The only surface difference is how the ephemeral token is authorized: candidate's POST /api/stt/token requires the user's Auth0 session, while public-site's POST /api/stt/token issues tokens to an anonymous identifier (cookie or IP+UA), so the fast path works for anonymous visitors too. The connection is prewarmed on first mic click (not on page load), so it does not open an OpenAI session for visitors who never dictate.

Architecture diagram

Mic tap
  ├─ getUserMedia (status: "preparing") — brief; every start, near-instant once permission granted
  ├─ [Realtime, both surfaces] prewarmed session ready?
  │     └─ yes: clone mic track into session
  │     └─ no:  this utterance uses batch only (silent fallback)
  └─ MediaRecorder.start() — status: "recording"
        │  (durable local capture — source of truth)
        ▼
     User taps ✓ / 10-min cap / byte cap
        │
        ├─ Realtime healthy the whole utterance?
        │     └─ yes: commit → await completed → insert text
        │     └─ no / any doubt: ▼
        └─ upload blob → POST /api/stt/transcribe
              └─ OpenAI gpt-4o-transcribe → insert text
              └─ failure → retain blob, offer Retry / Discard

Key modules

All shared UI lives in packages/ui (the @qwestly/ui library), consumed by both candidate and public-site.

Module	Location	Role
`useDictation`	`packages/ui/src/hooks/use-dictation.ts`	State machine. Owns capture, batch upload, retry, cap timer, and Realtime orchestration.
`RealtimeTranscriptionSession`	`packages/ui/src/lib/realtime-transcription-session.ts`	Persistent OpenAI Realtime WebRTC session. Prewarm, track swap, single-commit finalization, health tracking.
`RealtimeTranscriptBuffer`	`packages/ui/src/lib/realtime-transcription-session.ts`	Ordered per-item transcript accumulation for the Realtime path.
`PushToTalkButton`	`packages/ui/src/components/push-to-talk-button.tsx`	Mic button UI. Renders status-specific icon. Supports standalone mode (owns its own hook) or controlled mode (accepts a `dictation` instance).
`ChatInputBar`	`packages/ui/src/components/chat-input-bar.tsx`	Chat composer. Wires `useDictation({ realtime: true })`, renders waveform during recording, transcribing spinner, Retry/Discard row, and the checkmark flash on completion.
`AudioStreamBars`	`packages/ui/src/components/chat-input-bar.tsx`	The waveform visualization rendered during recording. Fed by `dictation.mediaStream`. This is the only live feedback during recording — there is no visible timer or caption panel.

API routes (app-specific)

Candidate app (transcribe: src/app/api/stt/[[...slug]]/route.ts via ApiRouter; token: src/app/api/stt/token/route.ts, a separate static route that takes precedence over the catch-all):

Endpoint	Auth	Purpose
`POST /api/stt/transcribe`	Auth0 session	Batch transcription. Accepts `multipart/form-data` with `audio` file + optional `language`. Forwards to OpenAI `gpt-4o-transcribe`. Returns `{ success: true, data: { transcript } }`.
`POST /api/stt/token`	Auth0 session	Ephemeral token for the Realtime fast-path. Returns a short-lived OpenAI Realtime API token with `turn_detection: null`.

Public-site (src/app/api/stt/transcribe/route.ts + src/app/api/stt/token/route.ts):

Endpoint	Auth	Purpose
`POST /api/stt/transcribe`	None (anonymous)	Batch transcription. Same shape as candidate but tighter rate limits (6/min per identifier vs 10/min per user) and identifier derived from agent session cookie or IP+UA hash.
`POST /api/stt/token`	None	Realtime token issuance with looser rate limits (20/min per identifier). Used by the prewarmed Realtime session.

Shared module (public-site/src/lib/stt/shared.ts):

Export	Purpose
`createSlidingWindowLimiter({ globalLimit, perIdentifierLimit })`	Factory for in-memory sliding-window rate limiters. Each route creates its own instance with separate caps.
`getClientIdentifier(request)`	Derives an anonymous identifier: `agent_session_id` cookie preferred, IP+User-Agent fallback.
`hashIdentifier(identifier)`	SHA-256 hash for the `OpenAI-Safety-Identifier` header, preventing raw IP leakage.
`validateLanguage(raw)`	Validates and normalizes a language code against the supported set (`en, fr, de, es, ja, ko, pt, zh`).

The recording lifecycle

State machine

idle → preparing → recording → transcribing → idle
  ↑                     ↓              ↓
  └─────────────────────┴──────────────┘
                      error

State	What the user sees	What's happening
`idle`	Mic icon	Nothing active.
`preparing`	Dimmed mic icon (brief; most noticeable on first-time permission)	`getUserMedia()` requesting mic access. Occurs on every start, but near-instant once permission is granted.
`recording`	Waveform visualization only (no timer)	`MediaRecorder` actively capturing. Chunks arrive every 1 second via `timeslice`.
`transcribing`	Spinner	Finalizing: either uploading the blob for batch transcription, or (Realtime fast-path) awaiting the `completed` event after the stop commit.
`error`	Error message + Retry/Discard buttons	Transcription failed but audio is retained in memory.

Stop reasons

Reason	Trigger	Toast shown
`user`	User taps ✓ or Enter	None — normal stop
`max-duration`	Timer reaches the configured cap (default 600s / 10 min)	"Reached the 10-minute dictation limit — transcribing what you said."
`max-size`	Recorded bytes reach 4 MB (`MAX_UPLOAD_BYTES`)	"Maximum recording length reached — transcribing what you said."
`cancel`	User taps ✕ or Escape	None — recording discarded

Byte-aware cap

The max-size stop reason is a defense against platform body limits. Vercel serverless functions reject request bodies over ~4.5 MB before the handler runs. The hook tracks total recorded bytes and auto-finalizes at 4 MB (MAX_UPLOAD_BYTES) — well under the platform limit even with multipart overhead and a trailing timeslice chunk.

This is format-adaptive: Chromium/Firefox (Opus ~240 KB/min) never hits the byte cap inside the 10-minute duration cap. Safari/iOS (AAC, higher bitrate) caps by size at ~4–5 minutes instead, with the same visible toast.

Two finalization paths

Batch path (always available)

MediaRecorder stops → chunks are assembled into a Blob
Blob is uploaded as multipart/form-data to POST /api/stt/transcribe
Server forwards the audio to https://api.openai.com/v1/audio/transcriptions with model gpt-4o-transcribe
Server returns { success: true, data: { transcript: "..." } }
Transcript is inserted into the chat input (appended with a space to preserve existing text)

Guarantees:

Always works — no dependency on a pre-established connection
Recording is never lost — blob is retained in memory on failure for retry
Server does not persist audio — forwarded to OpenAI in memory, no storage

Realtime fast-path (opportunistic, both surfaces)

On first mic click, a RealtimeTranscriptionSession begins connecting in the background (never awaited from the recording flow)
On subsequent recordings, if the session is ready, a clone of the mic track is swapped into the WebRTC sender via replaceTrack()
Recording proceeds normally via MediaRecorder (durable source of truth)
On stop, if the session was healthy the entire utterance, finalize() sends input_audio_buffer.commit and awaits the completed transcription event
If the session was never ready, had an error, or finalization times out → silent fallback to batch upload

Invariants:

Recording never waits for the Realtime session
The local MediaRecorder blob is always the source of truth — Realtime output is only used when provably complete
A per-utterance epoch fence prevents a cancelled utterance's late server events from leaking into the next utterance

Tradeoff: With turn_detection: null, transcription only starts at the commit (stop time), so finalization latency still scales with utterance length. This is acceptable for now; periodic commits are the documented future upgrade.

API endpoints

`POST /api/stt/transcribe`

Batch transcription for the capture-first flow.

Request: multipart/form-data

Field	Required	Description
`audio`	Yes	Audio file blob. Must be a supported MIME type with non-zero size.
`language`	No	ISO 639-1 code. Defaults to `en`. Must be in the supported set.

Supported audio types: audio/webm, audio/mp4, audio/x-m4a, audio/ogg, audio/mpeg, audio/wav

Candidate app limits: 15 MB max, 10/min per user, 100/min global, 120s upstream timeout, 300s function timeout. A Content-Length precheck rejects obviously oversized uploads before the body is buffered.

Public-site limits: 15 MB max, 6/min per identifier, 60/min global, 120s upstream timeout, 300s function timeout. Content-Length precheck rejects obviously oversized uploads before buffering.

Response (200):

{ "success": true, "data": { "transcript": "hello world" } }

Error responses:

Status	Code	Meaning
400	`INVALID_AUDIO`	Missing file, empty file, unsupported MIME type, or non-multipart body
401	`UNAUTHORIZED`	Candidate app only — no valid Auth0 session
413	`FILE_TOO_LARGE`	Audio exceeds max file size
429	`RATE_LIMITED`	Too many requests in the current window
500	`CONFIG_ERROR` (candidate) / `STT_NOT_CONFIGURED` (public-site)	`OPENAI_API_KEY` not set
502	`TRANSCRIPTION_UPSTREAM_FAILED`	OpenAI returned an error or network failure
504	`TRANSCRIPTION_TIMEOUT`	Upstream OpenAI call exceeded timeout

`POST /api/stt/token`

Ephemeral token for the Realtime WebRTC fast-path.

Request: POST /api/stt/token?language=en — language is a query param; there is no request body. Candidate sends the Auth0 session cookie; public-site is anonymous. The browser uses the returned token to open a WebRTC connection directly to OpenAI — no audio passes through this server.

Response (200): fields are top-level, not nested under data:

{ "success": true, "value": "<ephemeral-key>", "expires_at": 1712345678000 }

expires_at is milliseconds. Public-site additionally returns a session field; candidate does not. Only value is consumed by the client (RealtimeTranscriptionSession).

The session is created server-side with turn_detection: null (see Why turn_detection: null) — no server-side VAD, so transcription only starts when the client explicitly commits the audio buffer at stop.

Rate limits: candidate — 100/min global; public-site — 20/min per identifier + 100/min global.

Rate limits and safety

Candidate app (authenticated)

Per-user rate limiting using the Auth0 sub claim. 10 requests per user per minute, 100 global per minute. Windows are in-memory per-process and reset on deploy — sufficient for authenticated volume.

Public-site (anonymous)

Tighter limits on the unauthenticated surface:

Limit	Transcribe	Token
Per identifier	6/min	20/min
Global	60/min	100/min

Identifiers are derived from the agent_session_id cookie when present, falling back to a SHA-256 hash of IP:User-Agent. The hash prevents raw IP leakage in the OpenAI-Safety-Identifier header sent to OpenAI.

Caveat: On serverless deployments, each warm instance enforces its own window independently — real global throughput can exceed configured caps by roughly the number of concurrent instances. These are abuse backstops for current volume, not hard guarantees. Centralize in a shared store (Upstash/Redis) if sustained abuse is observed.

Error handling

Client-side errors (the hook)

Error code	Cause	Recovery
`UNSUPPORTED_BROWSER`	`getUserMedia` or `MediaRecorder` unavailable	Mic button hidden; text input unaffected.
`MICROPHONE_DENIED`	User denied permission or system blocked access	Error message shown. User must enable in browser/system settings and retry.
`MICROPHONE_NOT_FOUND`	No microphone device detected	Error message shown. Connect a microphone and retry.
`RECORDER_FAILED`	`MediaRecorder` emitted an error during recording	Error message shown. Retry starts a new recording.
`TRANSCRIPTION_FAILED`	Server returned an error or network failure	Audio blob is retained. Retry button re-uploads the same audio. Discard button drops it.

Too-short guard and partial-capture salvage

Two independent mechanisms (they govern different paths — don't conflate the thresholds):

Too-short guard (normal stop): a recording shorter than 500 ms (MIN_UTTERANCE_MS) is silently discarded — no upload, no error, no toast (accidental taps; also below the transcription API's minimum length). At or above 500 ms it transcribes normally.
Partial-capture salvage (recorder failure): if MediaRecorder errors during recording, the partial audio is still transcribed when at least 1 second (MIN_PARTIAL_CAPTURE_MS) was captured and chunks exist; otherwise a RECORDER_FAILED error is surfaced. This threshold applies only to the failure path, not to normal stops.

Server-side errors

All server errors return the standard { success: false, error: { code, message } } envelope. Upstream OpenAI errors are mapped to 502 (failure) or 504 (timeout) so the client can distinguish retryable vs permanent failures.

Environments and limits

Setting	Candidate (onboarding agent)	Public-site (anonymous agent)
Max recording duration	600s (10 min)	600s (10 min)
Byte-aware cap	4 MB	4 MB
Max file size (server)	15 MB	15 MB
Upstream timeout	120s	120s
Function timeout	300s	300s
Rate limit (per user/id)	10/min	6/min
Rate limit (global)	100/min	60/min
Authentication	Auth0 session	None (anonymous identifier)
Realtime fast-path	✅ Enabled (`realtime: true` in ChatInputBar)	✅ Enabled (same shared component)
Realtime prewarm	On first mic click	On first mic click
Realtime token auth	Auth0 session	Anonymous identifier (cookie / IP+UA)

Design decisions

Why capture-first, not realtime-first

The original implementation (use-streaming-stt.ts, removed) used WebRTC-only: mic tap → getUserMedia + token fetch + ICE + SDP negotiation → then start "recording." This took 1–2.5 seconds, during which the user was already talking — and a WebRTC track is live-only, so those first words were irrecoverably lost. A hard 120s silent cap and fixed post-commit timeout also truncated long recordings.

Capture-first fixes both by construction: local recording starts immediately, buffers everything, and can never clip or truncate. The Realtime path is layered on top purely as a latency optimization.

Why both paths

Batch alone fixes both original bugs and is the simplest correct thing. The Realtime path exists only to reduce post-stop latency. Two paths with a silent fallback is more complex than one, but the invariant "batch always works" keeps the complexity contained — failure in either path is recoverable.

Why no live transcript or visible timer

During recording, the UI shows only the waveform (AudioStreamBars). There is deliberately no live caption panel or countdown timer. A live transcript was considered but dropped because: (a) it created a misleading "the system is listening and understanding" impression before any transcription had actually happened; (b) it required the Realtime path to be working, coupling the UI to an optional optimization; (c) it added visual noise during a moment when the user should be focused on speaking. The waveform is the only live feedback. The formatTimer and elapsedSeconds facilities exist in the hook but are not surfaced in the current UI.

Why `turn_detection: null`

Primarily because the model requires it: gpt-realtime-whisper does not support turn detection — sending a turn_detection config returns a 400 ("Turn detection is not supported for this transcription model"). Server-side VAD simply isn't an available option, so the client must commit the audio buffer explicitly at stop.

This also aligns cleanly with the architecture: the local MediaRecorder is the source of truth for "recording ended," so an explicit client-side commit keeps the boundary unambiguous rather than letting the model guess. (A future live-caption or periodic-commit feature that wanted in-utterance VAD would need a VAD-capable streaming model such as gpt-4o-mini-transcribe.)

Platform body limit defense

Vercel serverless functions reject bodies over ~4.5 MB before the handler runs. The byte-aware cap (MAX_UPLOAD_BYTES = 4 MB) in the hook keeps all blobs under this limit regardless of browser encoder. The route's MAX_FILE_BYTES = 15 MB is an upper bound for the Content-Length precheck — in practice blobs never reach it because the hook finalizes at 4 MB.

Glossary

Term	Definition
Batch path	The primary finalization method: record locally, upload complete blob, transcribe via REST API. Always available.
Realtime fast-path	Optional WebRTC-based transcription using OpenAI's Realtime API. Lower latency when healthy, silent fallback to batch when not.
Capture-first	The architectural principle that local recording starts immediately and is the durable source of truth.
`MediaRecorder`	Browser API for recording audio from the microphone. Produces audio chunks via a timeslice.
`useDictation`	The React hook that owns the full recording → transcribe lifecycle.
`RealtimeTranscriptionSession`	A persistent OpenAI Realtime WebRTC session. Prewarmed on first mic click, track-swapped, single-commit finalization.
Epoch fence	A per-utterance generation counter that prevents a cancelled utterance's late events from leaking into the next utterance.
Byte-aware cap	The `MAX_UPLOAD_BYTES = 4 MB` limit that auto-finalizes a recording before the blob exceeds Vercel's platform body limit.
Timeslice	The interval at which `MediaRecorder` emits data chunks (1 second). Also the unit of the byte-aware cap.

_private/qwestly-docs/Features/dictation/dictation-system-overview.md