Configuration file

OpenGateLLM requires configuring a configuration file. This defines models, dependencies, and settings parameters. Playground and API need a configuration file (could be the same file), see API configuration and Playground configuration.

By default, the configuration file must be ./config.yml file.

You can change the configuration file by setting the CONFIG_FILE environment variable.

Secrets

You can pass environment variables in configuration file with pattern ${ENV_VARIABLE_NAME}. All environment variables will be loaded in the configuration file.

Example

models:
  [...]
  - name: my-language-model
    type: text-generation
    providers:
      - type: openai
        url: https://api.openai.com
        key: ${OPENAI_API_KEY}
        model_name: gpt-4o-mini

Example

The following is an example of configuration file:

# -------------------------------- dependencies ---------------------------------
dependencies:
  postgres: # required
    url: postgresql+asyncpg://${POSTGRES_USER:-postgres}:${POSTGRES_PASSWORD:-changeme}@${POSTGRES_HOST:-localhost}:${POSTGRES_PORT:-5432}/postgres
    echo: False
    pool_size: 5
    connect_args:
      server_settings:
        statement_timeout: "120s"
      command_timeout: 60

  redis: # required
    url: redis://:${REDIS_PASSWORD:-changeme}@${REDIS_HOST:-localhost}:${REDIS_PORT:-6379}
    max_connections: 200
    socket_connect_timeout: 5
    retry_on_timeout: True
    health_check_interval: 30
    decode_responses: False
    socket_keepalive: True

  elasticsearch: # optional
    index_name: opengatellm
    index_language: english
    number_of_shards: 1
    number_of_replicas: 0
    hosts: "http://${ELASTICSEARCH_HOST:-localhost}:${ELASTICSEARCH_PORT:-9200}"
    basic_auth:
      - "elastic"
      - ${ELASTICSEARCH_PASSWORD}

  # sentry:
  #   dsn: ${SENTRY_DSN}

  # langfuse:
  #   public_key: ${LANGFUSE_PUBLIC_KEY}
  #   secret_key: ${LANGFUSE_SECRET_KEY}
  #   base_url: http://localhost:3000

# ---------------------------------- settings -----------------------------------
settings:
  # disabled_routers: ["admin", "audio"]
  # hidden_routers: ["auth"]
  # usage_tokenizer: tiktoken_gpt2
  # app_title: My OpenGateLLM API

  # log_level: INFO
  # log_format: [%(asctime)s][%(process)d:%(name)s][%(levelname)s] %(client_ip)s - %(message)s

  swagger_version: 0.4.5
  # swagger_contact_url: https://github.com/etalab-ia/OpenGateLLM
  # swagger_contact_email: john.doe@example.com
  # swagger_docs_url: /docs
  # swagger_redoc_url: /redoc

  auth_secret_key: changeme
  auth_bootsrap_admin_username: admin
  auth_bootsrap_admin_password: changeme

  # rate_limiting_strategy: fixed_window

  # monitoring_sentry_enabled: True
  # monitoring_postgres_enabled: True
  # monitoring_prometheus_enabled: True

  # vector_store_model: my-model

  playground_opengatellm_url: ${OPENGATELLM_URL}
  # playground_opengatellm_timeout: 60
  # playground_disabled_pages: []
  # playground_default_model: my-model
  # playground_theme_has_background: True
  # playground_theme_accent_color: purple
  # playground_theme_appearance: dark
  # playground_theme_gray_color: gray
  # playground_theme_panel_background: solid
  # playground_theme_radius: medium
  # playground_theme_scaling: 100%
  # playground_swagger_url: http://localhost:8000/swagger
  # playground_reference_url: http://localhost:8000/redoc
  # playground_documentation_url: https://docs.opengatellm.org

# ----------------------------------- models ------------------------------------
# models:
#   - name: albert-testbed
#     type: text-generation
#     # aliases: ["model-alias"]
#     # owned_by: Me
#     # load_balancing_strategy: shuffle
#     # cost_prompt_tokens: 0.10
#     # cost_completion_tokens: 0.10
#     providers:
#       - type: vllm
#         url: http://albert-testbed.etalab.gouv.fr:8000
#         # key: sk-xxx
#         model_name: "gemma3:1b"
#         # timeout: 60
#         # model_hosting_zone: FRA
#         # model_total_params: 8
#         # model_active_params: 8

API configuration

Configuration file is composed of 3 sections, models:

models: to declare models API exposed to the API.
dependencies: to declare both required plugins for the API (e.g. PostgreSQL, Redis) and optional ones (e.g. Elasticsearch).
settings: to configure the API.

We don’t recommend to use the configuration file to declare models, prefer to use the API to declare models, by endpoints or on the Playground UI (see Models configuration).

Attribute	Type	Description	Default
models	array	Models used by the API. For details of configuration, see the Model section.	required
dependencies		Dependencies used by the API. For details of configuration, see the Dependencies section.	required
settings		For details of configuration, see the Settings section.	required

Model

In the model section, you define a list of models (routers and providers). These models are only used for the initial bootstrap of the API. The model section of the configuration is ignored if any models are already registered in the database.

Attribute	Type	Description	Default	Examples
name	string	Unique name exposed to clients when selecting the model.	required	gpt-4o
type	string	Type of the model. It will be used to identify the model type.	required	text-generation
aliases	array	Aliases of the model. It will be used to identify the model by users.	[]	[‘model-alias’, ‘model-alias-2’]
load_balancing_strategy	string	Routing strategy for load balancing between providers of the model.	shuffle	least_busy
cost_prompt_tokens	number	Model costs prompt tokens for user budget computation. The cost is by 1M tokens.	0.0	0.1
cost_completion_tokens	number	Model costs completion tokens for user budget computation. The cost is by 1M tokens. Set to `0.0` to disable budget computation for this model.	0.0	0.1
providers	array	API providers of the model. If there are multiple providers, the model will be load balanced between them according to the routing strategy. The different models have to the same type. For details of configuration, see the ModelProvider section.	required

ModelProvider

Attribute	Type	Description	Default
type	string	Model provider type.	required
url	null, string	Model provider API url. The url must only contain the domain name (without `/v1` suffix for example). Depends of the model provider type, the url can be optional (Albert, OpenAI).	None
key	null, string	Model provider API key.	None
timeout	integer	Timeout for the model provider requests, after user receive an 503 error (model is too busy).	300
model_name	string	Model name from the model provider.	required
model_hosting_zone	string	Model hosting zone using ISO 3166-1 alpha-3 code format (e.g., `WOR` for World, `FRA` for France, `USA` for United States). This determines the electricity mix used for carbon intensity calculations. For more information, see https://ecologits.ai	WOR
model_total_params	integer	Total params of the model in billions of parameters for carbon footprint computation. For more information, see https://ecologits.ai	0
model_active_params	integer	Active params of the model in billions of parameters for carbon footprint computation. For more information, see https://ecologits.ai	0
qos_metric	null, string	The metric to use for the quality of service policy. If not provided, no QoS policy is applied.	None
qos_limit	null, number	The value to use for the quality of service. Depends of the metric, the value can be a percentile, a threshold, etc.	None

Dependencies

Attribute	Type	Description	Default
elasticsearch	null	Elasticsearch is an optional dependency of OpenGateLLM. Elasticsearch is used as a vector store. If this dependency is provided, all documents endpoint are enabled. For details of configuration, see the ElasticsearchDependency section.	None
langfuse	null	See the LangfuseDependency section for more information. For details of configuration, see the LangfuseDependency section.	None
postgres		Postgres is a required dependency of OpenGateLLM to store API data. For details of configuration, see the PostgresDependency section.	required
redis		Redis is a required dependency of OpenGateLLM to store rate limiting counters and performance metrics. For details of configuration, see the RedisDependency section.	required
sentry	null	Sentry is an optional dependency of OpenGateLLM. Sentry helps you identify, diagnose, and fix errors in real-time. For details of configuration, see the SentryDependency section.	None

ElasticsearchDependency

Elasticsearch is an optional dependency of OpenGateLLM. Elasticsearch is used as a vector store. If this dependency is provided, all documents endpoint are enabled. Pass all arguments of elasticsearch.Elasticsearch class, see https://elasticsearch-py.readthedocs.io/en/latest/api/elasticsearch.html for more information. Other arguments declared below are used to configure the Elasticsearch index.

Attribute	Type	Description	Default	Examples
index_name	string	Name of the Elasticsearch index.	opengatellm	my_index
index_language	string	The language of the Elasticsearch index, composed by the value, the stopwords and the stemmer.	english	english
		For more information about stemmer, see https://www.elastic.co/docs/reference/text-analysis/analysis-stemmer-tokenfilter#analysis-stemmer-tokenfilter-configure-parms.
number_of_shards	integer	Number of shards for the Elasticsearch index.	12	4
number_of_replicas	integer	Number of replicas for the Elasticsearch index.	1	1

LangfuseDependency

Langfuse is an optional dependency of OpenGateLLM. Langfuse is used for LLM observability and tracing. In this section, you can pass all Langfuse client arguments, see https://python.reference.langfuse.com/langfuse for more information.

Attribute	Type	Description	Default	Examples
public_key	string	Langfuse public key.	required	pk-lf-…
secret_key	string	Langfuse secret key.	required	sk-lf-…
base_url	string	Langfuse server URL.	http://localhost:3000	http://localhost:3000

PostgresDependency

Postgres is a required dependency of OpenGateLLM. In this section, you can pass all postgres python SDK arguments, see https://docs.sqlalchemy.org/en/21/core/engines.html#engine-creation-apihttps://docs.sqlalchemy.org/en/21/core/engines.html#engine-creation-api for more information. Only the url argument is required. The connection URL must use the asynchronous scheme, postgresql+asyncpg://. If you provide a standard postgresql:// URL, it will be automatically converted to use asyncpg.

Attribute	Type	Description	Default	Values	Examples
url	string	PostgreSQL connection url.	required		postgresql+asyncpg://postgres:changeme@localhost:5432/postgres

RedisDependency

Redis is a required dependency of OpenGateLLM. Redis is used to store rate limiting counters and performance metrics. Pass all from_url() method arguments of redis.asyncio.connection.ConnectionPool class, see https://redis.readthedocs.io/en/stable/connections.html#redis.asyncio.connection.ConnectionPool.from_url for more information.

Attribute	Type	Description	Default	Values	Examples
url	string	Redis connection url.	required		redis://:changeme@localhost:6379

SentryDependency

Sentry is an optional dependency of OpenGateLLM. Sentry helps you identify, diagnose, and fix errors in real-time. In this section, you can pass all sentry python SDK arguments, see https://docs.sentry.io/platforms/python/configuration/options/ for more information.

No settings.

Settings

General settings configuration fields.

Attribute	Type	Description	Default	Examples
disabled_routers	array	Disabled routers to limits services of the API.	[]	[‘embeddings’]
hidden_routers	array	Routers are enabled but hidden in the swagger and the documentation of the API.	[]	[‘admin’]
app_title	string	Display title of your API in swagger UI, see https://fastapi.tiangolo.com/tutorial/metadata for more information.	OpenGateLLM	My API
routing_max_retries	integer	Maximum number of retries for routing tasks.	3
routing_retry_countdown	integer	Number of seconds before retrying a failed routing task.	3
routing_max_priority	integer	Maximum allowed priority in routing tasks.	4
usage_tokenizer	string	Tokenizer used to compute usage of the API.	tiktoken_gpt2
log_level	string	Logging level of the API.	INFO
log_format	string	Logging format of the API.	[%(asctime)s][%(process)d:%(name)s][%(levelname)s] %(client_ip)s - %(message)s
swagger_summary	string	Display summary of your API in swagger UI, see https://fastapi.tiangolo.com/tutorial/metadata for more information.	OpenGateLLM connect to your models. You can configuration this swagger UI in the configuration file, like hide routes or change the title.	My API description.
swagger_version	string	Display version of your API in swagger UI, see https://fastapi.tiangolo.com/tutorial/metadata for more information.	latest	2.5.0
swagger_description	string	Display description of your API in swagger UI, see https://fastapi.tiangolo.com/tutorial/metadata for more information.	See documentation	See documentation
swagger_contact	object, null	Contact informations of the API in swagger UI, see https://fastapi.tiangolo.com/tutorial/metadata for more information.	None
swagger_license_info	object	Licence informations of the API in swagger UI, see https://fastapi.tiangolo.com/tutorial/metadata for more information.	{‘name’: ‘MIT Licence’, ‘identifier’: ‘MIT’, ‘url’: ‘https://raw.githubusercontent.com/etalab-ia/opengatellm/refs/heads/main/LICENSE’\}
swagger_terms_of_service	null, string	A URL to the Terms of Service for the API in swagger UI. If provided, this has to be a URL.	None	https://example.com/terms-of-service
swagger_openapi_tags	array	OpenAPI tags of the API in swagger UI, see https://fastapi.tiangolo.com/tutorial/metadata for more information.	[]
swagger_openapi_url	string	OpenAPI URL of swagger UI, see https://fastapi.tiangolo.com/tutorial/metadata for more information.	/openapi.json
swagger_docs_url	string	Docs URL of swagger UI, see https://fastapi.tiangolo.com/tutorial/metadata for more information.	/docs
swagger_redoc_url	string	Redoc URL of swagger UI, see https://fastapi.tiangolo.com/tutorial/metadata for more information.	/redoc
auth_secret_key	null, string	Secret key for the API. It should be a random string with at least 32 characters. This key is used to encrypt user tokens, watch out if you modify the secret key, you’ll need to update all user API keys. If not provided, the master key will be used.	None
auth_bootsrap_admin_username	string	Username of the admin user created at the first startup.	admin
auth_bootsrap_admin_password	string	Password of the admin user created at the first startup.	changeme
auth_key_max_expiration_days	null, integer	Maximum number of days for a new API key to be valid.	None
auth_playground_session_duration	integer	Duration of the playground postgres_session in seconds.	3600
rate_limiting_strategy	string	Rate limiting strategy for the API.	fixed_window
monitoring_postgres_enabled	boolean	If true, the log usage will be written in the PostgreSQL database.	True
monitoring_prometheus_enabled	boolean	If true, Prometheus metrics will be exposed in the `/metrics` endpoint.	True
vector_store_model	null, string	Model used to vectorize the text in the vector store database. Is required if a vector store dependency is provided (Elasticsearch). This model must be defined in the `models` section and have type `text-embeddings-inference`.	None
document_parsing_max_concurrent	integer	Maximum number of concurrent document parsing tasks per worker.	10
front_url	string	Front-end URL for the application.	http://localhost:8501

Playground configuration

The following parameters allow you to configure the Playground application. The configuration file can be shared with the API, as the sections are identical and compatible. Some parameters are common to both the API and the Playground (for example, app_title).

For Plagroud deployment, some environment variables are required to be set, like Reflex backend URL. See Environment variables for more information.

Attribute	Type	Description	Default	Values	Examples
dependencies		Dependencies used by the playground. For details of configuration, see the Dependencies section.	required
settings		General settings configuration fields. Some fields are common to the API and the playground. For details of configuration, see the Settings section.	required

Dependencies

Attribute	Type	Description	Default	Values	Examples
redis	null	Set the Redis connection url to use as stage manager. See https://reflex.dev/docs/api-reference/config/ for more information. For details of configuration, see the RedisDependency section.	None

RedisDependency

Attribute	Type	Description	Default	Values	Examples
url	string	Redis connection url.	required		redis://:changeme@localhost:6379

Settings

Attribute	Type	Description	Default
auth_key_max_expiration_days	null, integer	Maximum number of days for a token to be valid.	None
routing_max_priority	integer	Maximum allowed priority in routing tasks.	10
app_title	string	The title of the application.	OpenGateLLM
playground_opengatellm_url	string	The URL of the OpenGateLLM API.	http://localhost:8000
playground_opengatellm_timeout	integer	The timeout in seconds for the OpenGateLLM API.	60
playground_disabled_pages	array	List of pages to disable from the navigation bar.	required
playground_default_model	null, string	The first model selected in chat page.	None
playground_theme_has_background	boolean	Whether the theme has a background.	True
playground_theme_accent_color	string	The primary color used for default buttons, typography, backgrounds, etc. See available colors at https://www.radix-ui.com/colors.	purple
playground_theme_appearance	string	The appearance of the theme.	light
playground_theme_gray_color	string	The secondary color used for default buttons, typography, backgrounds, etc. See available colors at https://www.radix-ui.com/colors.	gray
playground_theme_panel_background	string	Whether panel backgrounds are translucent: ‘solid’ \| ‘translucent’.	solid
playground_theme_radius	string	The radius of the theme. Can be ‘small’, ‘medium’, or ‘large’.	medium
playground_theme_scaling	string	The scaling of the theme.	100%
playground_swagger_url	null, string	Swagger URL. If not provided, deactivated swagger link in the navigation bar.	http://localhost:8000/docs
playground_reference_url	null, string	Reference URL. If not provided, deactivated reference link in the navigation bar.	http://localhost:8000/redoc
playground_documentation_url	null, string	Documentation URL. If not provided, deactivated documentation link in the navigation bar.	https://docs.opengatellm.org