A synthetic data generator and simulator for non-adversarial tasks is now available in public preview, accessible through the Azure AI evaluation SDK. One of the biggest evaluation challenges we hear from customers is that they do not have comprehensive, high-quality test datasets to run holistic evaluations. In March, we introduced an adversarial simulator, specifically designed to role-play with a user’s model or application to generate a high-quality test data for risk and safety evaluations. However, because it was designed to help accelerate adversarial red-teaming processes, it lacked the ability to simulate more general interactions with the actual target users of the application. Now, we are excited to announce an end-to-end synthetic data generation capability to help developers understand how their application responds to everyday user prompts. AI developers can use an index-based query generator and fully-customizable simulator to create robust test datasets around non-adversarial tasks and personas specific to their application. This can help organizations fill a critical gap in their existing evaluation toolkit, facilitating higher-quality evaluations and faster iteration on an application.
Learn more.