Optimize and compare agent prompts
Inspect one isolated 4+2+3 prompt experiment and its recorded evidence.
Optimize and compare agent prompts
Open Sidebar → Prompt Optimization. You can select an existing frozen pack or build one for a hotel agent: describe its business journeys, generate nine scenarios, inspect and edit them, run the independent review, then freeze the reviewed pack. Generation, review, and case regeneration make paid model calls. After an edit, save and review the changed scenarios before freezing; freezing locks the pack for experiments. To change a frozen pack, duplicate it into a new draft. Select the agent and frozen pack to start the experiment. A single experiment owns collection, optimization, final evaluation, reporting, cancellation, and recovery.
Experiment design
Every accepted pack freezes nine disjoint cases before execution:
- four training cases provide full V1 conversations and metric evidence to the optimizers;
- two tuning cases provide score-only evidence for candidate selection; and
- three hidden final cases remain unavailable until every optimizer arm is terminal and the available prompts are frozen.
The experiment runs native FAPO, native GEPA, and FAPO-guided GEPA independently. A failed arm remains visible and does not erase evidence from surviving arms. The final schedule pairs V1 and every successfully frozen candidate with each of the three final cases.
fapo_native is the Audex prompt-only pinned workflow used by this product. It is not the official FAPO coding-agent runtime or benchmark.
Reading a result
While work is active, the page shows the current phase and can request cancellation. Partial results remain visible after failure or cancellation. The evidence workspace has five views:
- Overview shows the current phase, coverage, actual usage, the three-by-four final evidence matrix, and the observed leader.
- Cases compares every recorded metric for one final case across V1, FAPO, GEPA, and guided GEPA. Select a score to inspect its recorded judge explanation, structured evidence, source run, and transcript.
- Prompts compares each frozen candidate with V1 as a line-oriented change view or as complete prompt text.
- Optimizers shows the recorded FAPO proposal history or GEPA candidate lineage and selection evidence for one arm at a time.
- Record shows the safe reproducibility record: identities and digests, case partitions, metric policy, participant model settings, configured limits, timestamps, failures, and actual usage.
The page loads the summary first. Prompt bodies, optimizer histories, run details, and the safe record load only when you open the corresponding view or result. While an experiment is active, open evidence views refresh when the durable run changes. If an older experiment did not record optimizer evidence, the page labels it Historical evidence unavailable instead of reconstructing a decision path that never existed.
The Cases view keeps missing metrics, invalid or failed runs, and run-load failures distinct. It never substitutes zero for unavailable evidence. A failed run load leaves the other arms usable and can be retried without reloading the case.
Final evidence is reported per case and arm. The observed leader is calculated from the available matched final evidence and remains informational even when coverage is incomplete. It is not a recommendation to deploy a prompt.
The cost summary comes from the actual provider-call journal. Known actual cost totals calls with a settled actual charge; unknown-cost calls counts journaled calls without one, including calls still in progress and terminal calls without resolved usage or cost. Unknown cost is never presented as zero.
Limits
- By default one experiment runs at a time. Starting another while one is active fails with an error; start it after the first one ends.
- A server restart or deployment ends running experiments as failed. Evidence recorded before the restart stays visible. To finish the work, start a new experiment; it runs from the beginning.
- Synthetic scenarios do not establish production performance.
- Adaptive conversations are stochastic even when the case and prompt identities are frozen.
- Failed or missing final slots reduce the evidence supporting the observed leader.
- A human transcript review is required before treating an experiment as proof of improvement.
- The workspace cannot apply or activate a prompt. Prompt application is outside this workflow.