OpenGateLLM documentation

Welcome to the OpenGateLLM official documentation! This comprehensive guide will walk you through everything you need to know to run, configure, and extend OpenGateLLM.

warning

The API is still under beta version, major breaking changes may occur.

What is OpenGateLLM?

OpenGateLLM is an open-source, production-ready API gateway, optimized for self-hosted models. It's designed to centralize, secure, and manage Generative AI access in a sovereign and cost-effective way.

OpenGateLLM addresses three critical challenges for organizations:

Cost control - Reduce expenses of commercial APIs and GPU infrastructure by using self-hosted models and build a mutualized infrastructure with your peers.
Data sovereignty - Keep sensitive data under your control
Privacy & security - No chat history storage, robust access control

Core principles

Open source and free forever - All features available without commercial licensing
High code quality - Built with maintainability and reliability in mind
Lightweight architecture - Focused feature set for optimal performance
High compatibility - Seamlessly integrates with GenAI ecosystem frameworks by OpenAI-compatible API
Production-ready - Engineered to handle high loads with advanced QoS features

OpenGateLLM architecture

OpenGateLLM is an alternative to...

Key features	OpenGateLLM	LiteLLM	TensorZero
🔌 OpenAI Compatibility	✅	✅	✅
📖 Open-source	✅	✅	✅
💻 Self-hostable	✅	✅	✅
💸 Free (all features)	✅	❌	✅
🏭 Support commercial and self-hosted models	✅	✅	✅
Account management
🎲 Playground UI	✅	✅	✅
👤 User management (API keys, budget...)	✅	✅	✅
🏢 Organization management	🚧	✅	❌
✏️ Project management	🔜	✅	❌
🔑 SSO support	🚧	✅	❌
High load features
⌛ Rate limiting	✅	❌	❌
⚡ Requests prioritization	✅	✅	❌
📈 Quality of service thresholds	✅	✅	❌
🔀 Model load balancing	✅	✅	❌
🔄 Model fallback	🔜	✅	❌
Monitoring & analytics
📊 Usage tracking	✅	❌	❌
📈 Carbon footprint	✅	❌	❌
🔗 Prometheus integration	✅	❌	❌
Privacy & security
🚫 No chat history storage	✅	❌	❌
🔒 Role-based access control	✅	❌	❌

Legend: ✅ supported — ❌ not supported — 🚧 work in progress — 🔜 in roadmap

Quickstart

Get started with OpenGateLLM in minutes with our quickstart guide here.

Community & support

OpenGateLLM is developed by Etalab since July 2024, the French government's open data and AI task force, with the support of CentraleSupélec.

This project exists thanks to all the people who contribute. OpenGateLLM thrives on open-source contributions. Join our community! Whether you're fixing bugs, adding features, or improving documentation, we welcome your help. Check out our Contributing Guide to get started.

Thanks to all contributors ❤️