Evals runs an Agent against saved test cases. Each case contains input, optional files, scoring criteria, and a passing percentage. Use the results to compare changes to the Agent's instructions, resources, or model.
Prepare representative cases
First, configure and try the Agent in Console. Include ordinary tasks and cases where the Agent should report missing evidence. Use fictional or approved test data.
- Open Agents → Evals.
- Create a new eval.
- Select the Agent.
- Name the eval.
- Add a case.
- Enter its input values and files.
- Add criteria that describe observable results.
- Set each criterion's points.
- Set the case's passing percentage.
- Save the case.
An eval can contain up to 50 cases. For document tasks, include criteria for factual accuracy, required content, and usable output files. For example: “The feedback cites evidence from the supplied essay.” Avoid a criterion such as “The answer is good.”
Run and inspect the eval
- Select Preview for an initial test with read-only workspace access.
- Choose Save & run for a new or changed eval, or Run all for a saved eval.
- Keep the tab open while it starts the cases.
- Open a result to inspect its output and criterion scores.
- Use Activity to inspect the Agent's work.
- Open each generated document.
- Compare it with the supplied evidence.
Run all starts cases in sequence. A case already started continues on the server if you leave the page. Stop cancels the active case and prevents the page from starting more cases. Live runs can perform the actions allowed by the Agent. Review those permissions before you choose Live.
Compare results
Use the case's latest result and average to find recurring weaknesses. Open history to compare individual results and grouped runs. Use the Preview and Live filters to compare the same execution mode.
The summaries cover the most recent 300 case executions. The page indicates when older results exist. Failed, stopped, ungraded, and judge-error results are separate from numeric score averages. A judge error is not a zero score. Deleted cases remain in history but leave the current case summary.
Each run retains its scoring criteria. Review those criteria when comparing results from different dates. An automated score supports your review; inspect important answers and documents yourself.
Continue with Agent logs and analytics.