Skip to content

Prometheus

OpenGateLLM can expose metrics in Prometheus format for monitoring and observability. This allows you to track API performance, usage patterns, and system health in real-time.

When Prometheus monitoring is enabled, OpenGateLLM exposes metrics at the /metrics endpoint in Prometheus format. These metrics can be scraped by a Prometheus server for visualization in tools like Grafana.

Once enabled, you can access the metrics endpoint at:

http://localhost:8000/metrics

This endpoint returns metrics in Prometheus text-based exposition format, which can be scraped by your Prometheus server.

All default metrics of prometheus-fastapi-instrumentator are available, see their README for more information. These metrics are prefixed by the namespace ogl_.

In addition, OpenGateLLM exposes the following metrics for inference:

Metric Type Description
ogl_inference_requests_total Counter Total number of LLM requests (endpoint, model, status_code).
ogl_inference_requests_duration_seconds Histogram Duration of LLM requests in seconds (endpoint, model, status_code).
ogl_inference_ttft_milliseconds Histogram Time to first token for streaming responses in milliseconds (endpoint, model, status_code).
ogl_inference_output_tokens_per_second Histogram Output generation speed in tokens/second (endpoint, model).
ogl_inference_tokens_total Counter Total number of consumed tokens with `type=prompt

Two ready-to-use Grafana dashboards are provided in the grafana/ folder of the repository:

Dashboard File Content
Inference grafana/inference.json LLM oriented metrics: request rate and success rate, request duration, time to first token, token consumption and output generation speed, all broken down by model and endpoint.
Traffic grafana/traffic.json API oriented metrics from prometheus-fastapi-instrumentator: request count, 2xx/5xx ratio, request duration percentiles, requests per second, process memory and CPU usage.

Both dashboards are filtered with template variables (Datasource, and Model, Endpoint and Percentile for the inference dashboard) so you can focus on a given model or endpoint.

To import a dashboard, in Grafana go to Dashboards > New > Import, upload the JSON file (or paste its content) and select your Prometheus datasource.

To enable Prometheus metrics exposure, you need to configure the monitoring setting in the settings section of your config.yml file.
Check Settings section in configuration file documentation for more information.

Example:

settings:
[...]
monitoring_prometheus_enabled: true

By default, Prometheus monitoring is enabled. Set it to false to disable the /metrics endpoint.

Configuration file documentation