We’re excited to announce expanded support for GPT4o models (including text and image-based capabilities) across all Generative AI policies in Azure API Management. With this release, you can now apply enhanced management and optimization features, including Token Limit Policy, Token Metric Policy, and Semantic Caching Policy, to GPT4o models deployed with Azure OpenAI.
Azure OpenAI Token Limit Policy
With this policy, you can set token-per-minute (TPM) limits for each API consumer, with options to assign limits based on keys like Subscription Key or IP Address. This policy also pre-calculates prompt tokens to avoid unnecessary requests to the Azure OpenAI backend when prompts exceed set limits.
Azure OpenAI Emit Token Metric Policy
This policy enables automatic token usage reporting to Azure Application Insights, capturing prompt, completion, and total token metrics. It also allows customizable dimensions, such as Subscription ID and IP Address, to help you analyze usage trends and optimize resource allocation.
Azure OpenAI Semantic Caching Policy
This policy utilizes semantic caching to store completions for prompts with similar meanings, using Azure Redis Enterprise or compatible RediSearch caches. The policy leverages the Azure OpenAI Embeddings model to identify and reuse semantically similar completions, saving tokens and enhancing response times.
To read more about this announcement, check out the blog post.