Mistral On-Prem
Mistral On-Prem is the solution of Mistral AI to self-hosted their commercial models.
See documentation
Supported models
Endpoints enabled by model type
Section titled “Endpoints enabled by model type”When you create a model router, you need to specify the model type. The model type determines the capabilities of the model and enables a specific set of endpoints.
| Model type | Endpoint enabled | Description |
|---|---|---|
text-generation | /v1/chat/completions | LLM models for text generation. |
text-classification | /v1/rerank | Reranking models for text classification. |
text-embeddings-inference | /v1/embeddings | Text embeddings models for text similarity and clustering. |
image-to-text | /v1/ocr | OCR models for image to text conversion (only Mistral On-Premise supported). |
image-text-to-text | /v1/chat/completions | Multi-modal models to chat and analyze images. |
automatic-speech-recognition | /v1/audio/transcriptions | Automatic speech recognition models for audio to text conversion. |
Supported self-hosted API providers
Section titled “Supported self-hosted API providers”Ollama
Ollama is a local-first model runtime for self-hosted models. To add it as a provider, you need to set openai as the provider type.
See documentation
Text Embeddings Inference
Hugging Face Text Embeddings Inference is an open-source API dedicated for embeddings and reranking. Is a great choice for self-hosted models.
See documentation
vLLM
vLLM is open-source production-grade LLM server. It supports a wide range of models and is a great choice for self-hosted models.
See documentation
WhisperX
WhisperX is a open-source framework to run Whisper with fast inference and a Pyannote diarization integration. Deploy it behind an OpenAI-compatible API and set openai as the provider type.
See documentation