New Microsoft tool lets devs spin up AI behavior tests using text descriptions
Scored daily by a customisable AI persona to surface the most relevant engineering leadership news.
Microsoft's open source framework for AI behavior testing is directly relevant to AI/ML evaluation pipelines.
Microsoft released ASSERT, an open-source framework that leverages LLMs to convert natural-language descriptions of policies and intended behaviors into structured, scored test suites for AI systems. It generates problem scenarios, runs them against the target system, and records intermediate tool calls and actions so developers can pinpoint failures—filling a gap left by generic benchmarks like HELM and AILuminate. The tool supports pre-deployment validation, post-deployment monitoring, and continuous regression testing for application-specific behaviors, such as limiting data access or email scope.