Expanded Agent Evaluations in Copilot Studio
Copilot Studio's agent testing now checks whether tasks got done, whether the right tools were used, harmful content, and response time, with reusable custom graders.
Details
- New measures include task completion, tool accuracy, and safety, with latency reported next to quality results
- The content safety test checks responses for hate, sexual content, violence, and self-harm against a threshold you set from level 0, the strictest, to level 7
- Evaluations now cover individual AI steps and wider automations, not only agent conversations
- A custom graders library lets you reuse graders across agents, and Copilot Studio can generate a grader for a specific agent, visible only inside your organization
- A new Evaluation Viewer role lets reviewers see results without broader maker permissions
Best for
Measuring an agent against a repeatable bar before and after every change.
When to use it
You're about to ship or change an agent and need evidence it still works and stays safe.
Who benefits
Makers, and the managers or reviewers who sign off on agents.
Join the newsletter to get ahead with AI.
I share strategies, new modules, free guides, and tools to get you better at using AI in your work.
Leading a team? See team programs →