Robustness and bias testing
Add controlled variants to see whether results hold across noise and accents.
Audio Robustness and Language Bias add comparable variants to eligible logical test definitions. Use them when you need to know whether an outcome holds beyond the base interaction.
Both controls are in the Real-World Reliability section of Evaluation Metrics.
Availability by run mode
| Run mode | Audio Robustness | Language Bias |
|---|---|---|
| Real Call | Available when eligible metrics and a noise type and level are available. | Available when at least two accents are configured and the bias requirements below are met. |
| Direct simulation | Available when eligible metrics and a noise type and level are available. | Available when at least two accents are configured and the bias requirements below are met. |
| WebSocket simulation | Available when eligible metrics and a noise type and level are available. | Not exposed. The Real-World Reliability heading describes Audio Robustness only. |
| Direct-channel agent | Not available. | Not available. |
| Transcript simulation | Available when eligible metrics and a noise type and level are available. | The current UI uses the local/direct configuration for this mode, so it is exposed when the bias requirements below are met. |
Audio Robustness is not restricted by the Language Bias mode predicate. It is active only after you select at least one available robustness-eligible metric; it also needs a usable noise type and non-None noise level.
Audio Robustness
Enable Audio Robustness to compare a clean interaction with a noisy one. Choose the Noise Type, Intensity Level, and Robustness Metrics that should be evaluated.
For each eligible definition, robustness produces two physical runs:
- one clean run;
- one noisy run.
Language Bias
Enable Language Bias to compare Accent A and Accent B for the selected Bias Metrics. It is available for phone/Real Call and local/direct configurations, not WebSocket. The control appears only when at least two accents are configured; to activate it, choose two distinct non-neutral accents and at least one available eligible metric. For each eligible definition, bias produces two physical runs, one for each accent.
How variants change run count
For each eligible logical definition:
- Base test: 1 run
- Audio Robustness only: 2 runs
- Language Bias only: 2 runs
- Audio Robustness and Language Bias: 4 runs
The expansion applies only when planned metrics overlap the robustness or bias metrics you selected. Available metrics and accents are configuration-dependent; the UI may indicate when no eligible metrics are configured.
Review Test runs needed before submitting. Afterward, use the suite’s Robustness Breakdown and Bias Breakdown to compare outcomes.