Audio Events And Traces
Audio workflows are event-heavy because they cross model output, provider calls, artifacts, playback, transcript projection, and thread history.
Use typed events for application behavior. Use traces and struct samples for local diagnostics.
Event Families
| Family | Examples | Use it for |
|---|---|---|
| User transcript events | transcript delta, completed, failed | Show or persist realtime input transcripts. |
| Assistant audio output events | output started, text pushed, segment, chunk, artifact, playback, completed, failed | Track final-text TTS and playback flow. |
| Tool/function events | tool start, args, result, end, custom tool progress | Understand realtime tool execution and audio-adjacent tool work. |
| Struct audio samples | playout, queue depth, underrun, diagnostic samples | Local low-overhead audio pipeline diagnostics. |
Assistant audio event type names are listed in the Events Reference. Prefer typed handling over hand-written JSON field assumptions.
Publication And Storage
Audio uses canonical events, process-local samples, content storage, and application storage according to the kind of data.
| Data | Usual destination |
|---|---|
| Transcript text | Thread history when committed. |
| Assistant text | Thread history as the primary assistant result. |
| Assistant audio artifact | IContentStore when artifact capture is enabled. |
| Playback progress | Canonical event when it belongs in replay; otherwise a trace or process-local sample. |
| Queue depth and underrun samples | Local struct-event observers or diagnostics. |
| Raw input audio | Explicit app storage only. |
This separation keeps the journal semantic while placing media bytes and high-frequency samples on storage surfaces designed for them.
Mixed Input Correlation
When a user message contains both text and audio, audio input metadata records the original content index for each detected media item. Use that index to correlate UI attachments, uploaded content, transcripts, and runtime metadata without guessing from display order.
Committed transcript metadata records the transcript text, provider key, route decision, topology, and response ownership when available. Assistant audio output events are separate: they describe synthesized or provider-owned assistant audio, not the user's input transcript.
Assistant Audio Output Flow
Final-text TTS emits events around the assistant audio output flow:
- the assistant text is available,
- synthesis starts,
- segments or chunks may be produced,
- an artifact may be stored,
- playback may queue, progress, complete, or fail,
- the output flow completes or falls back to text-only.
Playback truth is intentionally conservative. A queued artifact is not proof that the user heard the audio.
Struct Samples
Audio struct samples are process-local diagnostics. They are useful when code in the same process needs low-overhead observations such as playback queue depth or underruns.
They are distinct from AgentEvent publication. Publish a canonical AgentEvent or content artifact when a hosted client or replay projection must observe the semantic result.
Testing
For event-driven audio tests:
- subscribe before
RunAsync, - capture typed events into a list,
- assert event type, session id, thread id, response id, output flow id, and ordering,
- test text-only fallback paths as well as successful synthesis,
- test playback truth separately from synthesis truth.