💰** AI usage and cost tracking ** Token spend from AI sessions and the AI chat is now recorded per workspace, user, provider and model. ➡️ Admins see the whole workspace over the last 7, 30 or 90 days, grouped by day, user or model. ➡️ Every member can see their own usage. ➡️ Costs marked with ~ are estimates. A figure the provider billed back is shown as is, and a model with no known rate reads as unpriced. 📖 https://www.windmill.dev/changelog/ai-usage-and-cost

Fratera · 6h ago

🚀 AI agent evals (beta) A reusable AI agent could only ever run as a flow step, and there was nowhere to keep the inputs that exposed a problem in production. Evals fix that. **You can now measure an agent against a dataset of cases, score every answer, and diff two runs case by case.** ➡️ A dataset is a set of cases, each a question and an expected answer. It belongs to the reusable ai_agent resource, so it outlives the flow step and two runs of the same agent stay comparable. Up to 1000 cases per dataset. ➡️ A run is one flow job: a parallel loop over the cases, each answer scored by one branch per scorer. Scorers are your own code, or an LLM judge that returns a score and a reason. The run outlives the tab that started it. ➡️ Every run is kept, so you can answer the question that matters after a change: did it get better, or did one case break. Compare any two runs of a dataset and the difference shows up case by case and scorer by scorer. This is a beta, tell us what breaks. 📖 Read more: https://www.windmill.dev/changelog/ai-agent-evals

Fratera ·