Copilot Studio Launched #261

Improvements to agent Evaluations in Microsoft Copilot Studio

Makers get more insight into agent evaluation results

Microsoft has made improvements to the Evaluations experience available in Microsoft Copilot Studio. They help makers assess agent quality and behavior with less manual effort.

Key points

  • Richer explanations help makers understand why evaluations succeed or fail.
  • Reasoning traces and cited knowledge sources help explain agent behavior and grounding.
  • Evaluation run comparison helps identify improvements and regressions across runs.
  • Evaluations support datasets with up to 100 test cases.
  • Makers can generate datasets from configured knowledge sources.

What to do

No action is required. Tell makers about evaluation explanations, run comparison and the 100-test-case dataset limit. Highlight customizable dataset and question generation, single-response datasets, and improved checks before evaluations run.

Timeline

  1. 21 October 2026 Published
Speaker notes

These improvements give makers more ways to understand agent quality rather than just see whether an evaluation succeeded or failed. Reasoning traces and cited knowledge sources help explain agent behavior and grounding. Comparing evaluation runs helps makers spot both improvements and regressions. Dataset tools include customizable dataset and question generation, generation from configured knowledge sources, and support for single-response datasets. Improved validation, connection prerequisite checks, credit-limit enforcement and actionable error messages help makers resolve configuration issues before running evaluations. Microsoft states that the feature is now available and that no action is required; no licensing requirements are specified.

Sources and links

Area, status, change type, licence and flags are indicative. Always open the source before you act on a card.