Risk and safety evaluations for indirect prompt injection attacks are now available in public preview, accessible through Azure AI Foundry UI and SDK experiences. Indirect prompt injection attacks (also known as cross-domain prompt injection attacks or XPIA) are an emerging attack vector where a threat actor poisons a model’s grounding data source, such as a public website, email, or internal document, to pass hidden, malicious instructions to a model and circumvent safety guardrails. With the Azure AI Evaluation SDK, users can now simulate indirect prompt injection attacks on their generative AI model or application and measure how often their AI fails to detect and deflect the attacks (the defect rate) along subcategories of manipulated content, intrusion, and information gathering. Users can also drill into evaluation details to better-understand how their application typically responds to these attacks and the associated risks. With this information, users may decide to activate Prompt Shields using Azure AI Content Safety, adjust grounding data sources, or apply other mitigations in their system message before rerunning the evaluation and deploying to production.
Learn more.