AI agent evals (beta)A reusable AI agent could only ever run as a flow step, and there was nowhere to keep the inputs that exposed a problem in production. Evals fix that.
You can now measure an agent against a dataset of cases, score every answer, and diff two runs case by case.
A dataset is a set of cases, each a question and an expected answer. It belongs to the reusable ai_agent resource, so it outlives the flow step and two runs of the same agent stay comparable. Up to 1000 cases per dataset. A run is one flow job: a parallel loop over the cases, each answer scored by one branch per scorer. Scorers are your own code, or an LLM judge that returns a score and a reason. The run outlives the tab that started it. Every run is kept, so you can answer the question that matters after a change: did it get better, or did one case break. Compare any two runs of a dataset and the difference shows up case by case and scorer by scorer.This is a beta, tell us what breaks.
Read more: https://www.windmill.dev/changelog/ai-agent-evals
