Microsoft’s new ASSERT framework lets developers test AI behavior using plain English

Microsoft has released a new open-source framework called ASSERT that aims to simplify how developers test whether their AI systems behave as intended. Instead of writing complex code for each evaluation scenario, the tool allows engineers to describe desired behaviors in plain English, and then automatically generates test cases, runs them, and scores the results.

What ASSERT does differently

ASSERT, short for Adaptive Spec-driven Scoring for Evaluation and Regression Testing, is designed to fill a gap that broader, more general AI benchmarks cannot address. While industry-wide evaluations like Stanford’s HELM or MLCommons’ AILuminate measure model capabilities at scale, they often miss application-specific nuances. For example, a document research AI agent might need to follow company-specific policies about emailing external contacts or sharing confidential data with executives. ASSERT lets developers define those rules in natural language, and the framework generates targeted tests to check compliance.

Sarah Bird, chief product officer of Responsible AI at Microsoft, said that evaluations are critical for making informed decisions about AI deployment. “If you don’t understand the behavior of the AI system, it’s really hard to know if it’s meeting your organization’s bar,” Bird said. She noted that ASSERT can be used during development, after deployment, and for continuous monitoring, making it a practical tool for production environments.

How the framework works

The framework takes a plain-language description of expected behavior and policies, then converts it into a structured set of acceptable and unacceptable actions. From there, it generates problem scenarios and test cases, runs them against the target system, and scores the results. Developers can also inspect the intermediate steps and tool calls the AI system made, which helps pinpoint where failures occur.

For instance, a developer might specify that an AI assistant should not send emails to people outside the company, should limit confidential information to C-level executives, and should provide concise summaries that account for prior context. ASSERT would then create test cases to verify each of those rules on an ongoing basis.

Why this matters for AI safety

The release comes at a time when the AI industry is increasingly focused on repeatable testing and regression checks. As models become more capable, ensuring they behave reliably in specific contexts has become a priority. Tools like ASSERT help bridge the gap between general model evaluation and the real-world constraints of a product or service. This is especially relevant for enterprises deploying AI in regulated industries, where compliance and safety are non-negotiable.

Conclusion

Microsoft’s ASSERT framework represents a practical step toward making AI behavior testing more accessible and thorough. By allowing developers to define expectations in natural language and automating the evaluation process, it addresses a growing need for application-specific testing that goes beyond generic benchmarks. As AI adoption accelerates, tools that simplify safety and compliance checks will become increasingly valuable.

FAQs

Q1: What does ASSERT stand for?
A: ASSERT stands for Adaptive Spec-driven Scoring for Evaluation and Regression Testing. It is an open-source framework from Microsoft.

Q2: Can ASSERT be used for continuous monitoring?
A: Yes, Microsoft says ASSERT can be used during development, after deployment, and for continuous monitoring of AI systems.

Q3: How does ASSERT differ from other AI evaluation tools?
A: ASSERT focuses on application-specific behavior testing using natural language descriptions, while broader benchmarks like HELM or AILuminate measure general model capabilities. ASSERT fills the gap for context-specific, policy-driven evaluations.