🧪

What is AI Gateway?

AI Gateway is an enterprise-grade gateway for managing AI models, tools, and agents, powered by Azure API Management. It provides security, performance, observability, cost control, and extensibility (including MCP protocol support) for production AI applications. The repository offers 30+ hands-on Jupyter notebook labs, Bicep infrastructure templates, and APIM policies.

How to use AI Gateway?

Clone the repository, install Python 3.12+ and uv, open the notebooks in VS Code with the Jupyter extension, and follow step-by-step instructions. Each lab deploys to your Azure subscription using provided Bicep templates and APIM policies.

Key features of AI Gateway

• 🔐 Security via OAuth 2.0, managed identities, and content safety filtering
• ⚡ Performance with load balancing, semantic caching, and request routing
• 📊 Observability through token metrics, built-in logging, and tracing
• 💰 Cost control using rate limiting, quota management, and FinOps framework
• 🔌 Extensibility with MCP protocol support, function calling, and multi-model routing
• 🏛️ Labs follow Azure Well-Architected Framework pillars (security, reliability, performance, operations, cost)

Use cases of AI Gateway

• Manage and control access to Large Language Models with enterprise-grade policies
• Enable secure tool access using MCP protocol and function calling capabilities
• Build and control agentic applications with orchestration frameworks
• Implement token rate limiting, semantic caching, and backend pool load balancing
• Automate failover between AI models using circuit breaker and retry patterns

FAQ from AI Gateway

What dependencies and runtime requirements does AI Gateway need?

You need Python 3.12+, uv (Python package manager), VS Code with Jupyter extension, an Azure subscription with Contributor + RBAC Administrator roles, and the Azure CLI authenticated to your subscription.

Where does AI Gateway data live?

All infrastructure and data are deployed to your own Azure subscription. No data leaves your controlled environment.

What authentication and authorization mechanisms are supported?

AI Gateway supports OAuth 2.0, managed identities, and client authorization flows for MCP tools.

What is AI Gateway's known limitation?

The README does not explicitly state limitations. Labs assume an Azure subscription and may incur costs; cleanup notebooks are provided.

How does AI Gateway compare to other AI gateway alternatives?

AI Gateway is built on Azure API Management, offering a centralized governance layer for security, safety, cost controls, resiliency, observability, and governance across models, tools, and agents.