Voice Agent
Understand results

Review an ASR benchmark replay

Compare frozen provider results, inspect transcript and passage-matching evidence, and understand what can and cannot be recalculated.

The ASR Workbench compares speech-to-text provider outputs without treating provider agreement as ground truth. Use it to inspect transcription quality, execution evidence, and downstream impact for a reviewed report.

Open the published results view

Published Results is the read-only view for a deployment-approved provider-matrix report. It includes every provider configuration and retained measurement in that artifact. Use its filters to select provider lanes, corpus, category, WER basis and minimum, ITN misses, reference conflicts, text matches, and sort order. Filtering changes only the presentation; it does not recompute or edit results.

Accounts with the report_viewer role can access only this view, their session details, and logout. Access also requires a verified email whose exact domain is allowed by the deployment. Published Results does not provide report upload, ground-truth editing, audio playback, provider execution, or administrative controls.

Open the right view

The ASR Workbench section of the sidebar contains five related views:

  • Published Results opens the deployment-approved, read-only provider-matrix report described above.
  • Plan explains the Tier A standalone, Tier B contact-center, and Tier C end-to-end evaluation boundaries.
  • Benchmark compares providers by call and exposes metric, execution, transcript-delta, and evidence details.
  • Evaluation Suite aggregates calls over the suite timeline and lets you narrow the same cohort by provider, metric, or case.
  • Rubrics & Metrics explains comparison terminology and directs you back to an explicitly loaded Workbench report for its in-memory metric definitions. It does not read a report catalog from browser storage.

Inside Evaluation Suite, start with By model for the primary ASR comparison, use By agent for the current agent-suite context, and open By call for audio, transcript, and run-level evidence.

Controls can be hidden or disabled by the deployment's capability profile. A replay-only profile can inspect an existing report but cannot execute paid provider or metric work. If the capability check is temporarily unavailable, the workbench keeps operator controls disabled, labels the connection state separately from deployment-disabled, and offers up to two manual retries.

Read the quality and timing columns

Use the metric direction before comparing values:

MeasurementInterpretation
WER and CERWord and character error rates. Lower is better, and both require a human-reviewed reference.
Entity recallShare of declared reference entities recovered by the transcript. Higher is better; unavailable reference entities produce an unavailable result, not zero.
Entity-WERWord error rate restricted to explicitly reviewed entity occurrences. Lower is better. It requires an authoritative transcript plus annotation values/forms and normalized-token spans. Missing, mismatched, nested, or overlapping spans produce Not measurable.
PNERWord error rate restricted to reviewed occurrences whose entity has the proper_noun scoring role. Lower is better. A call with no explicit proper-noun annotation is Not measurable, not 0%.
I-WERWord error rate on inverse-text-normalization targets such as spoken “three” and written 3. Lower is better. Each target requires the explicit itn role, a written value, a scorer-distinct spoken form, and its reviewed normalized-token span. Missing, mismatched, ambiguous, or overlapping pairs produce Not measurable.
AudEx meanArithmetic mean of compatible measured AudEx scores retained for that provider. It is a compact comparison, not a new canonical metric or an overall provider grade.
Mean ASRMean provider-reported ASR processing time per call. It excludes downstream agent and metric processing and is not end-to-end latency.
RTFProvider processing seconds divided by source-audio seconds. Values below 1 mean processing completed faster than the recording duration.
Time to first token (TTFT)Milliseconds from the first audio frame to the first non-empty returned text, whether interim or final. Lower is better. It requires retained packet timing from a real-time-paced streaming run; batch, unpaced, or text-free runs are Not measurable.
Interim update latencyMedian caller-observed delay for non-initial interim caption updates. Lower is better. It requires a real-time-paced stream with retained interim packet timing; a final-only stream has no applicable interim samples and is Not measurable.
Finalization latencyMedian delay from the echoed endpoint audio packet to the finalized transcript. Lower is better. It requires a real-time-paced stream with retained final packet timing; a run with no usable final sample is Not measurable.

Open a provider row or Evidence and calculation to inspect the measurements behind an aggregate. Missing prerequisites remain explicit as Not measurable, Not applicable, or Reference unavailable.

Specialized metrics show transcript authority and semantic-annotation authority separately. Their effective authority is never stronger than the weaker input. An unreviewed annotation draft has annotation mode and effective authority NONE, even when its transcript was human reviewed. Workbench labels that metric Preview only and not publishable. A verified script annotation can be publishable at VERIFIED_SCRIPT; a human-approved annotation can retain human-reviewed authority. The metric detail shows transcript provenance, annotation review status and reviewers, effective authority, and publication state as separate fields. Preview-only values remain available under the separate Inspect preview-only metrics section, but they do not contribute to quality charts, measurement coverage, family or provider aggregates, provider comparisons, Best labels, or the evaluation matrix. Published CSV output keeps them out of metrics.csv and places their retained evidence in preview-metrics.csv.

When evidence is missing, follow the action shown by the Workbench: attach or approve the semantic annotation manifest for entity metrics; add explicit proper_noun or itn roles where applicable; review exact normalized-token occurrences for repeated phrases; resolve nested/overlapping spans; or rerun a real-time-paced streaming capture when packet timing is absent. Provider processing lag remains unavailable when the provider did not explicitly send the field—an absent scalar is never displayed as measured zero.

In Evaluation Suite, canonical ratio values are displayed as percentages for readability. This is presentation only: for example, a retained ratio of 0.154 appears as 15.4% and the replay value is not rewritten.

Review transcript passage matching

Set Audio language or locale when submitting Ink Whisper audio, for example fr-FR. The Workbench requires an explicit supported language so an unspecified locale cannot silently become English. When Ink Whisper returns word offsets, the Workbench groups words into shorter timed segments so stereo dialogue can be ordered by time.

Live deterministic ASR accepts one to five distinct provider configurations. A one-provider run is ASR-only: it preserves that transcript and reports passage matching as Not run because there is no provider pair to compare. Transcript passage matching and Hybrid mode require two to five providers and compare each selected provider with every other selected provider. Matching shows which passages appear to correspond, which passages have no accepted counterpart, and which results remain unresolved. It helps prioritize listening when no human reference exists; provider agreement is not treated as truth and the view does not identify the correct transcript.

When Cartesia Ink 2 is enabled by the deployment, the Workbench lists it as an optional English ASR provider. It sends each mono channel at about recording speed and limits recordings to 120 seconds per channel. Stereo recordings require an explicit speaker layout. Cartesia publishes a rate of 3 credits per audio second per channel, including silence; the Workbench shows this as guidance, not an invoice. Ink 2 transcript text is retained exactly. On stereo recordings, the report retains each channel's known TA or AUT role without claiming a chronological merge. Ink 2 has no timed speaker segments, so a run cannot be submitted with both Ink 2 and AudEx metrics.

Cartesia Ink Whisper is a separate complete-file provider. It accepts WAV for the usual Workbench channel layouts, or mono FLAC when Ink Whisper is the only selected provider and Source audio is selected. Audio must use 8–192 kHz PCM with an 8, 16, 24, or 32-bit sample width. Its Workbench limit is 3,600 seconds per channel and 100 MB per file. The run stays in the Workbench's background job while Cartesia handles one HTTP transcription request; Cartesia does not expose a submit-and-poll job for this endpoint. Ink Whisper can return word timestamps; if requested timestamps are absent, the provider result fails instead of silently showing untimed words. Neither Cartesia provider has a verified spoken/written formatting or ITN switch in this Workbench.

Deterministic is the safe default matching mode. It compares exact evidence locally and never sends transcript text for semantic adjudication. Select Hybrid matching explicitly when external transcript processing is approved for the run. Hybrid mode retains deterministic evidence and can use external semantic adjudication for eligible non-identical passages. If external processing is unavailable or a proposal cannot be used, the report keeps deterministic evidence and records unresolved content instead of inventing a match.

Read the matching summary fields independently:

  • Completion reports whether matching produced a complete, partial, or not-run result. It is execution coverage, not an accuracy score.
  • Reliability reports the experimental validation state. A complete result can still have experimental, unvalidated reliability.
  • External adjudication needed and attempted distinguish demand from execution. The Workbench does not display a semantic-validation verdict.

Choose one provider pair at a time in Provider pair. Comparable pairs appear first and comparisons with unusable input appear in a separate group at the bottom. When present, Parakeet ↔ Soniox is selected first as a useful speaker-label-versus-timing comparison; every other pair remains selectable. The selected comparison reports its completion and sequence relation, then shows aligned provider passages in one shared scrolling reader. Each stored segment is formatted as a turn: AUT and TA appear only when that provider supplied the speaker label, while missing speaker or timing evidence remains explicitly unavailable. Exact highlights and timestamp ranges stay attached to the provider segment that supplied them. Legacy unverified intervals retain a visible timing-uncertainty label instead of being presented as exact provider timing. The canonical verbatim transcripts remain available in the Verbatim provider output disclosure and are never rewritten to manufacture turn or timing metadata. Aligned rows follow the left provider's passage order, so one-sided or uncertain rows can appear outside the right provider's native order. This is called out explicitly for reordered results, and the full transcript columns remain the authority for each provider's native order. Passage groups use these outcomes:

  • Matched contains passages accepted as corresponding.
  • No match appears on the empty side of a one-sided result; it does not declare either provider wrong.
  • Could not match reliably preserves unresolved evidence without converting uncertainty into a verdict.
  • Not compared — input unusable explains when the source evidence could not be compared.

The provenance badge distinguishes Deterministic exact, Empty-side evidence, LLM-assisted match, and Unresolved evidence results. The compact full-transcript view computes bounded character highlights in the browser only to make the two retained strings easier to scan; those highlights are not accuracy evidence or backend localization. When an unusually divergent passage exceeds that display budget, the Workbench labels a conservative changed span instead of attempting an unbounded exact diff. The expanded per-turn evidence uses only backend Unicode code-point offsets for trusted word-level highlights. When backend localization is unavailable, the passage remains visible with an explanation such as Word-level differences unavailable at an ambiguous boundary; the browser does not generate or repair those word deltas. If a provider's matching input is unusable, the reader shows both retained immutable transcripts without highlights and explains that alignment and timestamp mapping are unavailable.

The All turns, Needs review, and Differences controls narrow only the expanded per-turn evidence. The synchronized full transcript remains complete so a filter cannot hide surrounding conversational context.

A passage accepts timing only from a complete, valid interval supplied by the backend; absent or invalid intervals display Timing unavailable. A valid interval appears as a timestamp range together with any retained legacy or uncertainty qualifier. It becomes a seek control when audio is available and remains a read-only timestamp when audio is unavailable. Legacy workbench-v1 substitution, insertion, and deletion cards remain visible as lexical evidence with Passage matching: Not run, but they never infer timing or offer disagreement seeking. Ordinary timed provider transcript segments can still seek the attached recording.

Passage matching is display-only in workbench-v2 and workbench-v3. Inspect the retained groups, provenance, uncertainty, and timing evidence, but do not treat the browser as a verdict or correction editor. There is no matching review workflow, and exported workbench-v2 reports do not carry a browser-authored verdict, review subject, or corrected grouping.

Follow a current run

Use the three current-run views for different questions:

  • Overview summarizes matching completion, reliability, and whether external adjudication was needed or attempted. These fields describe coverage and execution, not correctness.
  • Call detail shows the exact provider-pair groups. Read Matched, No match, Could not match reliably, and Not compared — input unusable before using audio seeking or backend word highlights to inspect a passage.
  • Execution keeps Transcript matching usage separate from AudEx / provider metric usage. Matching reports logical requests, the SDK-send ceiling, observed sends and retries, queue wait, provider latency, and provider-reported token totals. At least N is a lower bound, while Unknown means the observed total is incomplete with no known amount; neither is an exact zero.

The loaded complete report remains in memory only for the current tab. The Workbench does not save complete reports, report transcripts, or report-projected metric-catalog text in local or session storage. Logout, authorization failure, or a workspace change clears the visible report, uploaded run inputs, and active run tracking. Use Export report before leaving the tab if you need to keep an explicitly downloaded copy; importing that copy later does not create an automatic browser cache. Complete exported or imported reports can contain private transcripts and evidence, so keep downloaded copies only in approved storage and share them only with authorized recipients.

The standalone Rubrics & Metrics page also deletes any legacy browser catalog cache when it mounts and never renders it. Inspect report-specific families, prerequisites, formulas, and availability states only inside the Workbench tab where you explicitly loaded that report.

Attach local audio safely

The tracked commit-safe replay contains no audio content, filename, or local path. A replay may also omit the source-audio identity entirely; in that case the workbench marks playback as unavailable instead of offering an attachment control.

When a replay retains a source-audio SHA-256 and authenticated audio replay is configured for your profile, choose Load server audio to fetch the tenant-scoped frozen or completed-run recording. Large recordings are not downloaded merely by opening the view, and a failed load can be retried. Frozen files are checked against the requested SHA-256; completed-run requests are checked against the immutable digest captured when the upload was accepted. The server does not fall back to another tenant or an unverified file.

The deployment-wide private replay is available only to super administrators. Tenant administrators and members use tenant-scoped replay when it is configured for their workspace, or select the matching local recording instead. The browser attaches a local file only after confirming that its SHA-256 matches the report case. A mismatched or unverifiable file remains detached. The tracked commit-safe replay has neither a server recording nor a retained audio identity, so audio stays unavailable there.

Understand recalculation limits

Adding or editing a human reference can recalculate eligible reference-backed transcript metrics in the loaded browser report. Select one measured metric when you want to refresh only that local result. The change is applied in the current tab only; export the report if you need to keep it.

Local reference-metric recalculation does not call an ASR provider, place a call, or update a stored production run. AudEx recalculation is a separate paid-evaluator path for eligible legacy reports: the Workbench shows the expected and maximum evaluator-call counts and requires explicit confirmation before it runs. The commit-safe demo removes private transcripts, reference text, entity values, and raw judge evidence, so its frozen scores cannot be independently recalculated.

Use the frozen nine-call demo correctly

The bundled demo contains nine reviewed long-call cases and five provider/model configurations: Deepgram, OpenAI, Parakeet, Soniox, and Voicea. The separate five-call Swiss pilot is not part of this tracked cohort.

Token, infrastructure, evidence, audio, and USD sections appear only when retained data and the active access profile allow them. Hidden data is omitted rather than shown as a misleading zero or a restricted score card.

When USD visibility is enabled but the replay has no complete pricing snapshot, the dashboard shows Not priced. It does not infer a dollar amount from token counts alone.

The Suite can also show explicitly unlinked downstream demo evidence. For example, Red Alert results for agents outside the nine-call cohort carry a not linked disclaimer and do not contribute to ASR or AudEx aggregates.

On this page