Metrum AI Router Adds Quality-Aware Routing for On-Prem and Cloud Model Deployments

Summary

Metrum AI Router is an Apache-2.0 gateway for platform teams that need to route agent and application workloads across private, hosted, and hybrid model targets. Its model groups define quality, capability, validation, and cost contracts; request-shape checks remove targets that do not support the requested API dialect, modalities, tools, structured outputs, reasoning controls, or output caps before a provider call. The router also admits estimated input, tool, schema, and output tokens before cache-miss calls, reserves in-flight quota, and reconciles completed usage. Its Learned Routing Policy (LRP) is trained offline with operator-owned outcomes using LightGBM, per-target quality and output-token models, isotonic calibration, and shared feature and embedding artifacts. Among already eligible targets, LRP chooses the cheapest predicted model that meets the configured quality floor, or the highest predicted-quality target when none qualifies; uncertainty can trigger abstention to an anchor or fallback. A synthetic holdout of 113 requests reported cost of $0.1507895 for LRP versus $0.151829 for the always-anchor policy, with a quality mean of 1.0 and no floor violations, but the project marks the result as non-promotable because cost and real-data gates failed and embeddings and outcomes were synthetic. Operators are expected to collect and label traffic, train and validate bundles, run at least a 24-hour shadow period, and promote through an explicit shadow/enforce mode with baseline rollback. The router supports Linux binaries, Docker Compose, and Kubernetes, can mix OpenAI-compatible private targets with hosted providers, and includes PII filtering, safe policy context, request evidence, usage reports, caching, fallbacks, and Claude Code and Codex paths. The repository also documents limitations: provider uptime, model quality, compliance, and capability claims remain operator responsibilities; SQLite is unsuitable for horizontally scaled replicas; and OpenAI Responses or cross-dialect streaming may require unary upstream processing.