Make an AI voice feature start playing in seconds, not after a long wait

---
name: voice-ai-pipeline
description: Turn AI-generated text into voice audio that starts playing in seconds instead of making users wait for the whole file — chunked/progressive TTS delivery, pause fidelity across voice model tiers, loudness normalization, and a failure ladder that degrades to a proven batch fallback instead of erroring. Use when an LLM or dynamic script needs to become spoken audio on a live, latency-sensitive path.
---

# Implement: a streaming/progressive text-to-speech pipeline

You are helping a developer turn AI-generated (or any dynamic) text into voice
audio that starts playing quickly, instead of the user waiting for the whole
piece to render. Implement the pattern below IN THEIR codebase, adapting to
their language, framework, TTS provider, and audio player. Do not copy any
single vendor's SDK blindly — the pattern is provider-neutral; wire it to
whatever they have.

## Before you write code, detect their stack

Ask or infer: their language/runtime, their TTS provider (ElevenLabs, Azure
Speech, Google Cloud TTS, AWS Polly, OpenAI TTS, or none yet), whether that
provider offers both a fast/cheap tier and a higher-quality tier, its
documented concurrency limits, and what their client-side audio player can
already do (does it support gapless multi-track queueing, or only a single
file?). Adapt every step to what you find. If they have no player capable of
multi-track playback, either recommend the smallest viable upgrade or scope
down to the plain-batch path (skip steps 1–5, do 6–9 only).

## Build these, in this order

1. **Pick the delivery shape.** Default to progressive multi-track delivery
   if the client can (or can cheaply be made to) drain a queue of audio
   chunks gaplessly: return after the first chunk is ready, keep
   synthesizing + uploading the rest in the background. If the client can
   only play one file, do plain batch (skip to step 6) — don't force a
   client rewrite to unlock this recipe.

2. **Split text into synthesis chunks at natural seams.** Break at
   author-placed pause markers first, not by raw word count. Tune the target
   chunk size against the SPECIFIC tier's measured per-request latency: too
   few chunks puts one slow chunk on the critical path; too many creates a
   straggler tail that blows budget on a slow tier. See
   `reference/chunk-and-pace.ts` for the chunking + pause-classification
   shape.

3. **Translate pauses per model tier, never assume one syntax works
   everywhere.** Classify each pause as shallow (ride inline, using whatever
   native pause syntax that tier honors) or deep/authored (make it REAL
   reinjected silence spliced in at a chunk boundary — synthesize a short
   silent clip and stitch it in). Verify by ear on EACH tier you support; do
   not assume a script that paces correctly on one tier paces the same on
   another.

4. **Tier the voice model to the surface.** Fast/cheap tier for the
   interactive, someone's-waiting path. Slower/higher-quality tier ONLY for
   anything pre-generated ahead of need (a cron, a "prepare while the user
   is still browsing" trigger). Respect the provider's concurrency ceiling
   as a semaphore shared across ALL concurrent work (live + background); on
   a rate-limit response, retry with backoff instead of failing the whole
   generation.

5. **Give every synthesis call an explicit, strongly-held timeout chained to
   a caller abort.** Do not rely on a composite/derived AbortSignal alone —
   in some runtimes an unreferenced derived timer can be garbage-collected
   and silently never fire, hanging the request and starving a shared
   concurrency slot. Hold your own timer, clear it in a `finally`.

6. **Concatenate by re-encoding, never raw byte-copy.** Byte-copy concat is
   faster but leaves audible seam glitches (frame-boundary misalignment) and
   wrong duration metadata (a player reads only the first chunk's header).
   Re-encode through the format's real encoder in the same pass — still
   sub-second for a multi-minute piece.

7. **Fold loudness normalization into the same re-encode pass.** One filter
   pass, no added latency, targeting a standard loudness level with a
   true-peak ceiling so output volume doesn't drift piece to piece. See
   `reference/chunk-and-pace.ts` for where this slots into the concat step.

8. **Build the failure ladder, and make every rung detectable.** Name each
   failure point (upstream generation stalls, TTS connect fails, TTS returns
   no audio within budget, mid-stream error) with a wall-clock detection
   budget and a fallback action. The last rung is always a plain,
   already-proven batch path — never new code written only for the failure
   case. See `reference/failure-ladder.md` for the table shape.

9. **Instrument the seams.** Log time-to-first-audio-byte, per-chunk
   synthesis time, total stream duration, and which fallback rung (if any)
   fired — per generation, not just a single total.

## Guardrails to enforce as you build

- The interactive path never uses the expensive/slow tier live — only
  pre-generated ahead of need.
- A single hung synthesis request cannot hold a shared concurrency slot
  forever — verify the timeout actually fires under a simulated hang.
- Concatenation always re-encodes; never leave a fast byte-copy path in the
  code even "temporarily."
- Every fallback rung actually degrades to something that works — verify by
  forcing each failure point (bad API key, injected timeout, truncated
  response) and confirming the user still gets audio.
- Compression middleware / edge caching is explicitly disabled on the
  streaming route — verify bytes arrive incrementally with a raw client, not
  just "it played in a browser tab."

## Verify before done

Simulate each rung of the failure ladder (upstream stall, TTS connect
failure, TTS timeout, mid-stream error) and confirm the user still gets
audio via the batch fallback. Confirm a deep authored pause survives
playback on every voice tier in use. Confirm two consecutive generations of
the same script land at the same perceived loudness. Report what you wired
and what you stubbed.

## Read alongside

`RECIPE.md` (the why + the honest receipts) and `reference/` (working code
to adapt, not copy verbatim).

/**
 * reference/chunk-and-pace.ts — provider-neutral shape for splitting a
 * script into synthesis chunks, classifying pauses, and re-assembling the
 * final audio with correct pacing and consistent loudness.
 *
 * This is a pattern to ADAPT, not a drop-in library. Swap the TTS call, the
 * silence-synthesis call, and the concat/loudnorm call for your own
 * provider's SDK / your own audio toolchain. No vendor-specific code here —
 * see RECIPE.md steps 2-3 and 6-7 for the reasoning behind each shape.
 */

// ── 1. Parse the script into text blocks + standalone pause markers ───────

type ScriptItem =
  | { type: 'text'; text: string }
  | { type: 'pause'; seconds: number }

/**
 * Split a script on blank lines into ordered items: spoken text blocks and
 * standalone `<pause seconds="N"/>`-style markers the author placed between
 * passages. Adapt the marker syntax to whatever your authoring pipeline
 * uses (SSML <break/>, a custom tag, plain "..." conventions).
 */
function parseScript(script: string): ScriptItem[] {
  const blocks = script.split(/\n\s*\n/).map((b) => b.trim()).filter(Boolean)
  return blocks.map((b) => {
    const m = b.match(/^<pause seconds="(\d+(?:\.\d+)?)"\s*\/?>$/i)
    return m ? { type: 'pause', seconds: parseFloat(m[1]) } : { type: 'text', text: b }
  })
}

// ── 2. Group into chunks, deciding which pauses become REAL silence ───────

interface Chunk {
  rawText: string
  /** Seconds of REAL reinjected silence to splice in after this chunk. */
  silenceAfterSec: number
}

/**
 * Group text blocks into at most `maxChunks` synthesis chunks, choosing the
 * `maxChunks - 1` LARGEST standalone pauses as real chunk-boundary silence.
 * Every other pause (smaller, or falling inside a merged chunk) gets
 * re-inlined as a token for the per-tier intra-chunk transform to handle
 * natively (see step 3 below) — it still paces, just not as spliced silence.
 *
 * Tune `maxChunks` and your own target-words-per-chunk against the SPECIFIC
 * tier's measured per-request latency (RECIPE.md step 2) — a size that's
 * right for a fast tier creates a straggler tail on a slow one.
 */
function groupIntoChunks(items: ScriptItem[], maxChunks: number): Chunk[] {
  type Base = { rawText: string; pauseAfterSec: number }
  const base: Base[] = []
  for (const it of items) {
    if (it.type === 'text') base.push({ rawText: it.text, pauseAfterSec: 0 })
    else if (base.length > 0) base[base.length - 1].pauseAfterSec += it.seconds
    // a leading pause with nothing to attach to is dropped — nothing to hold silence before
  }

  if (base.length <= maxChunks) {
    return base.map((b) => ({ rawText: b.rawText, silenceAfterSec: b.pauseAfterSec }))
  }

  const boundaryCount = base.length - 1
  const keep = Math.max(1, maxChunks - 1)
  const chosen = new Set(
    Array.from({ length: boundaryCount }, (_, i) => i)
      .sort((a, b) => base[b].pauseAfterSec - base[a].pauseAfterSec || a - b)
      .slice(0, keep),
  )

  const chunks: Chunk[] = []
  let parts: string[] = []
  for (let i = 0; i < base.length; i++) {
    parts.push(base[i].rawText)
    const isLast = i === base.length - 1
    if (isLast) {
      chunks.push({ rawText: parts.join('\n\n'), silenceAfterSec: 0 })
      break
    }
    const gap = base[i].pauseAfterSec
    if (chosen.has(i)) {
      chunks.push({ rawText: parts.join('\n\n'), silenceAfterSec: gap })
      parts = []
    } else if (gap > 0) {
      parts.push(`<pause seconds="${gap}"/>`) // re-inlined for the intra-chunk transform
    }
  }
  return chunks
}

// ── 3. Per-tier intra-chunk transform: what a pause becomes INSIDE a chunk ─

interface VoiceTier {
  id: string
  /** Does this tier honor a literal pause marker natively, or does it need
   *  translation to inline cue tags ("[short pause]" style)? */
  honorsNativePauseMarker: boolean
  /** Stability/consistency setting. Raise this for any tier that synthesizes
   *  chunks INDEPENDENTLY with no cross-chunk context — low stability lets
   *  the voice "wander" (accent, persona) chunk to chunk (RECIPE.md: accent
   *  drift failure mode). */
  stability: number
}

function intraChunkText(rawText: string, tier: VoiceTier): string {
  if (tier.honorsNativePauseMarker) {
    // Tier reads the marker natively — leave it alone.
    return rawText
  }
  // Tier needs translation to inline cue tags it actually understands.
  return rawText.replace(/<pause seconds="(\d+(?:\.\d+)?)"\s*\/?>/gi, (_, sec) =>
    parseFloat(sec) <= 1.5 ? ' [short pause] ' : ' [long pause] ',
  )
}

// ── 4. Synthesize chunks, bounded by the provider's concurrency ceiling ───

/** Run `fn` over `items` in waves of at most `concurrency`, preserving order. */
async function runInWaves<T, R>(
  items: T[],
  concurrency: number,
  fn: (item: T) => Promise<R>,
): Promise<R[]> {
  const out: R[] = []
  for (let i = 0; i < items.length; i += concurrency) {
    const wave = items.slice(i, i + concurrency)
    out.push(...(await Promise.all(wave.map(fn))))
  }
  return out
}

/**
 * Synthesize one chunk with an OWN, strongly-referenced timeout chained to a
 * caller abort signal. RECIPE.md step 5: a composite/derived AbortSignal with
 * nothing holding the timeout strongly can silently never fire in some
 * runtimes, hanging the request and starving a shared concurrency slot.
 */
async function synthesizeChunk(
  text: string,
  tier: VoiceTier,
  opts: { signal?: AbortSignal; timeoutMs: number; ttsCall: (text: string, tier: VoiceTier, signal: AbortSignal) => Promise<Buffer> },
): Promise<Buffer> {
  const ac = new AbortController()
  const onExternalAbort = () => ac.abort()
  if (opts.signal) {
    if (opts.signal.aborted) ac.abort()
    else opts.signal.addEventListener('abort', onExternalAbort, { once: true })
  }
  const timer = setTimeout(() => ac.abort(), opts.timeoutMs) // explicit, strongly held
  try {
    return await opts.ttsCall(text, tier, ac.signal)
  } finally {
    clearTimeout(timer) // always clear — this is what makes the timeout GC-safe
    if (opts.signal) opts.signal.removeEventListener('abort', onExternalAbort)
  }
}

// ── 5. Reinject real silence at chosen boundaries, concat, normalize ──────

/**
 * Interleave chunk audio with reinjected silence, then concatenate by
 * RE-ENCODING (never raw byte-copy — RECIPE.md step 6) and fold loudness
 * normalization into that same encoding pass (step 7). `concatAndMaster`
 * is a stand-in for your own audio toolchain (e.g. an ffmpeg invocation
 * with a concat demuxer + a loudnorm filter in one pass).
 */
async function assembleFinalAudio(
  chunks: Chunk[],
  chunkAudio: Buffer[],
  deps: {
    makeSilence: (seconds: number) => Promise<Buffer>
    concatAndMaster: (buffers: Buffer[], loudnessTargetLufs: number, truePeakDb: number) => Promise<Buffer>
  },
  loudnessTargetLufs = -16,
  truePeakDb = -1.5,
): Promise<Buffer> {
  const interleaved: Buffer[] = []
  for (let i = 0; i < chunkAudio.length; i++) {
    interleaved.push(chunkAudio[i])
    const gap = chunks[i].silenceAfterSec
    if (gap > 0 && i < chunkAudio.length - 1) {
      // Stretch slightly and clamp — a proven pacing policy, not a magic number
      // you need to invent from scratch. See RECEIPTS-internal.md for the
      // exact multiplier and clamp used at origin.
      interleaved.push(await deps.makeSilence(gap))
    }
  }
  return deps.concatAndMaster(interleaved, loudnessTargetLufs, truePeakDb)
}

export { parseScript, groupIntoChunks, intraChunkText, runInWaves, synthesizeChunk, assembleFinalAudio }
export type { ScriptItem, Chunk, VoiceTier }

# Reference: the failure ladder (RECIPE.md step 8)

A shape to adapt, not a fixed list — name every real failure point in YOUR
pipeline, give each a wall-clock detection budget, and give each a fallback
action. The bottom rung is always a plain path that already works.

| Point | Detection | Action |
|---|---|---|
| Upstream text generation stalls before any output | error/no-first-token event | abort this attempt, fall back to batch |
| Upstream text generation emits nothing within budget | wall-clock timer | abort, fall back to batch |
| TTS connection fails to open within its connect budget | connect timeout | switch to the provider's plain HTTP/single-shot mode, if one exists |
| TTS returns no audio within budget from stream start | wall-clock timer | abort, fall back to batch |
| Mid-stream error from either upstream | error event | abort and fall back — UNLESS enough audio has already reached the client that finishing cleanly (a replayable partial) beats restarting from zero |
| Client-side stalled read | player/transport error | client-side retry against the plain batch endpoint |
| A synthesis request hangs past its own timeout | the per-chunk timeout in `reference/chunk-and-pace.ts` | abort that specific chunk, free its concurrency slot immediately — do NOT let the whole generation wait on it |
| Provider returns a rate-limit response | HTTP 429 / equivalent | retry with backoff bounded by the same request's timeout, do NOT immediately dump the whole generation to the fallback tier |

## The rule this table encodes

Every row needs BOTH a detection (how do you know this failed, and how
quickly) and an action (what happens instead). A row with detection but no
tested action is a failure mode waiting to become an incident. A row with
neither is invisible until a user reports it.

The last rung — plain batch fallback — must be a path that is ALREADY
proven and already running for some traffic (e.g. the pre-streaming
implementation you had before this recipe). Never write new "just in case"
fallback code that itself has never been exercised in production; an
untested fallback is not a fallback, it's a second failure mode.