How Microsoft Evaluates Copilot Without Touching Customer Data

Enterprise assistants need enterprise-shaped test data, but real enterprise data is exactly what you cannot use. Microsoft Research’s post From synthetic enterprise data evaluations to real Microsoft Copilot AI confidence describes the Industry Evaluation Data Generation Platform. Numbers below are Microsoft’s own, from the blog, and not independently verified.

Three pillars

1. Synthetic enterprise environments. Full-scale simulated tenants with realistic users, meetings, emails, documents and org structures, no customer data. They are time-evolving, with meeting histories and email threads developing over weeks, because many Copilot queries are time-aware (“What did the team decide last Thursday?”).
2. Automated query generation. A pipeline that turns generic workplace queries into industry-specific ones.
3. Structured quality frameworks. Checks that the synthetic data is good enough to evaluate against.

Grounding

To keep tenants realistic, generation is grounded in public sources that Microsoft says were reviewed by its legal and policy organisation: O*NET occupational data (roles, activity mix, workflows), online job postings (titles, hierarchies, departments) and SEC filings (scale, business units, financial context). Three industries were generated: pharmaceutical (regulatory submissions, clinical trial discussions), capital markets (deal pipelines, compliance, portfolio reviews) and IT/software (sprint planning, code review, launches).

The query pipeline

The blog describes six stages in prose: a large task library, narrowing to the industry, domain adaptation, AI generation grounded in the tenant, humanising rewrites, and validation. The stage names in the diagram are my paraphrase.

Quality and safety

Quality is scored on structural fidelity, contextual relevance and behavioural realism, plus a content-safety check for harmful or personally identifiable material and impersonation of real people. Reported results: a 39% improvement in the composite quality metric against the initial approach, some industry environments scoring above 0.9 of 1.0, hundreds of users per environment, and evaluation dataset creation moving from a multi-day manual task to automated runs, with estimated savings of about $100,000 per dataset. The platform is published to an internal catalog as the first standardised evaluation baseline for Copilot.

What I take from it

• Reproducibility is the quiet win. Regenerating a dataset from the same configuration and getting identical results is what lets you compare releases. Manual dataset creation makes reproducibility and scaling harder.
• Synthetic data removes the review bottleneck. Microsoft reports its teams can reuse these approved datasets without extra privacy-review overhead, which it says makes broad adoption possible.
• Time-awareness matters for agents. Static fixtures miss a whole query class. If your agent reasons over history, your test environment needs history.
• A caveat. The blog reports quality against its own internal scoring. It does not show how well scores on synthetic tenants predict satisfaction on real ones, which is the claim the title promises. Treat it as a method to borrow, not proof of the outcome.

For agentic systems the blog notes the same approach is aimed at moving from single-turn Q&A to multi-step workflows, where grounded, repeatable environments matter even more.

Source: Microsoft Research blog.