From nightly benchmark to ranked answer.
Model Maverick is a measurement system with a policy engine on top. Here is the whole pipeline — how scores come to exist, how your rules shape them, and why every answer can be audited after the fact.
The capability matrix
The core data structure is a matrix: model columns on one
axis, capability × difficulty cells on the other. A model
column is a (provider, model, endpoint, settings) tuple — so
claude-sonnet-5 direct and the same model with thinking
enabled are separate, separately measured things. Each cell holds
quality, pass rate, cost, and latency — smoothed over recent nights of
runs (more on that below).
The matrix is fed by a private, versioned task corpus:
200+ tasks across ten capabilities and five
authored difficulty tiers. Golden answers and hidden test suites never
enter a prompt, which blocks the contamination that makes public benchmark
scores unreliable. Every score row and every recommendation carries a
matrix_version, so any decision can be audited against the
exact tasks and graders that produced it.
Grading: deterministic first, judges last
Every task carries its grader as data, validated at authoring time. Deterministic graders (test suites, schema validation, rule scans) and reference graders (golden answers, key-fact coverage) do most of the work. A single cross-family LLM judge — temperature 0, order-swapped to cancel position bias — covers rubric dimensions like code idiom and summary readability.
The judge role itself is earned empirically: the
critique_and_judging capability doubles as judge selection, so
judging goes to models that measurably rank well at it, not to reputation.
Scoring: variance is risk, so it subtracts
Each task runs three times per model column. A model that passes four times in five is riskier than its average suggests, so variance subtracts from quality. A second penalty, ambiguity, applies when graders disagree about a single output — an answer the graders split on is harder to trust than a clean result with the same score.
Nightly numbers feed an exponentially weighted moving average (EWMA) — a running average where the most recent nights count the most and older results gradually fade. Think of it as a trend line rather than a scoreboard: last night moves it, but can't yank it. Rankings are driven by that smoothed score, so one noisy night cannot flip a recommendation, while a genuine change still shows up within a few nights. And when the smoothed score drops sharply, that raises a regression alert — which is how you find out a provider changed a model under you before your users do.
Predictions locked before results, scored after
Before each nightly run, the system locks its predicted ranking and stores it. After the run, the prediction is scored against reality with a Brier score. Locking first makes hindsight impossible: the calibration series shows exactly how well the benchmark predicts itself, capability by capability, and where it is blind. That history — not optimism — is what gates any shortcut like matrix completion.
What happens every night.
Five stages, in order — ending in the capability matrix your requests read the next day.
The policy engine: five steps, in order
- Pool restriction. Intersect all model columns with the caller's pool. Pools are the compliance boundary — a request can never widen its own.
-
Allowlist. If
allowed_modelsis present, it must be a subset of the pool. An unknown entry is a loud, named error — no silent dropping, no silent widening. - Standing constraints. Apply stored filters: zero-retention requirements, cost and latency ceilings. These can't empty the set, because configuration writes reject any combination with zero usable columns.
- Request preferences. Per-request ceilings deprioritize rather than drop. If every survivor violates one, the best available returns anyway with the violations named.
- Rank. Raw scores normalize across the survivors at query time — "best of these four" is computed on the four — then order under your mode: quality, balanced, or cost.
The result is a guarantee: a well-formed request from a configured tenant always returns a model. Difficulty shortfalls and preference violations arrive as flags on a real recommendation, never as an empty response.
Classification: declared or inferred
A declared workload (capability, difficulty,
domain) skips classification entirely and sends nothing but
labels. A raw prompt goes through a small, cheap classifier at temperature
0 with structured output, cached by prompt hash, with its confidence
surfaced in the response. High-stakes routes can require declared workloads
and take the classifier out of the loop.
Tie-breaking favors your invoice
Ties inside the noise band resolve to the cheaper column, and a same-family downgrade is preferred over a cross-family swap — fewer prompt-format surprises when you step down.
Where the cost numbers come from
Every model column carries a current price snapshot — provider list prices, synced as they change. Every benchmark run records the tokens it used and what it cost. Fold the two together and each matrix cell knows the measured, typical cost of work at that capability and difficulty: what a tier-3 extraction actually costs to run on a given model, at today's prices.
That number enters the decision three ways. Your per-capability weights
say how much cost matters next to quality and latency. Your mode sets the
band — balanced takes the cheapest model within a point or two of
the best; cost widens the band. And a per-request ceiling like
max_cost_per_run is checked against the estimate: a pick that
violates it still returns — deprioritized, never dropped — with the
violation named, limit and estimate side by side, so your code can decide.
The estimate is honest about being an estimate: measured from corpus tasks of the same capability and difficulty, priced at current rates. It's the typical cost of work shaped like yours — not a metered quote on your exact document.
What happens on every request.
One call in, a ranked answer out, milliseconds end to end — and a paper trail that lasts. On an SDK, all of it happens invisibly inside the client you already use.
"Summarize this customer email in one sentence."
classified: summarization_synthesis · tier 1
Every tracked model clears tier-1 summarization, so the cheapest column wins. A frontier model here would do the same job at 30× the price.
then: gemini-3.1-flash · escalation, same pool
"Migrate our billing service from REST to gRPC — schemas, handlers, and tests."
classified: code_generation · tier 5
Only frontier columns clear tier-5 code generation at this quality bar. Sending this to a mini model saves pennies and costs you a sprint.
then: claude-fable-5 · escalation, same pool
Same endpoint. Same policy. The prompt decides the model — and when the nightly benchmarks shift, so does the answer.
Never pay more for an answer than it's worth.
Every feature earns something each time it runs — a ticket triaged, an
email summarized, a contract parsed. A model call that costs more than
that turns the feature into a leak. One number in your policy,
max_cost_per_run, is the cap on what a single call may
cost. Three walkthroughs:
The free tier
A free-plan email summarizer. It earns nothing directly — it exists to convert signups.
Maverick can only surface models whose measured cost for this work fits under the cap. Your free tier physically can't choose a model that loses you money.
The high-volume route
Tagging 50,000 support tickets a month. Easy work — tier 1 — and every tracked model passes it.
Same pass rate either way. The guardrail turns "we should really downsize that model" from a backlog ticket into a number the engine enforces — and $1,400 a month stays yours.
The route worth real money
Extracting obligations from signed contracts — each run replaces an hour of paralegal review.
A missed obligation costs more than any model call, so here the quality bar leads and the guardrail stays out of the way. Guardrails are per route — tight where calls earn pennies, generous where mistakes are expensive.
And when nothing fits under a ceiling? A per-request cap never strands you: the best model still returns, with the violation named — limit and estimate, side by side — so your code decides. A standing tenant cap is a hard wall, and Maverick refuses to save a wall that would leave you zero usable models.
One request, one defensible answer.
The engine's native surface — what every SDK, gateway plugin, and MCP
tool speaks underneath, and an open door for stacks we don't wrap yet.
REST with per-tenant keys; agents get the same three operations as MCP
tools (recommend_model, get_policy,
list_models).
POST /v1/recommend
{
"workload": {
"prompt": "Extract the parties, dates, and obligations from this contract: ..."
},
"pool": "vertex-only",
"mode": "balanced",
"constraints": { "max_cost_per_run": 0.01 }
}{
"classification": {
"capability": "structured_extraction",
"difficulty": 3,
"domain_tags": ["legal"],
"high_stakes": true,
"confidence": 0.91
},
"recommendations": [
{ "provider": "anthropic", "model": "claude-sonnet-5", "endpoint": "vertex",
"score": 94.1, "clears_difficulty": true,
"reason": "cheapest column clearing tier-3 extraction at your quality bar" },
{ "provider": "anthropic", "model": "claude-opus-4-8", "endpoint": "vertex",
"score": 96.3, "clears_difficulty": true, "reason": "escalation target" },
{ "provider": "google", "model": "gemini-3.1-pro", "endpoint": "vertex",
"score": 92.8, "clears_difficulty": true, "reason": "cross-family alternative" }
],
"check_spec": { "type": "json_schema", "ref": "checks/extraction/v3" },
"constraints_met": true,
"violations": [],
"recommendation_id": "rec_01J...",
"matrix_version": "2026-07-01",
"measured": true
}Every decision leaves a trace
Each response carries a recommendation_id. The by-id history
endpoint returns the full decision trace: classification with confidence
and method, the pool and allowlist applied, mode and weights, every
surviving column with its query-time score, and every excluded column with
the reason it never competed. "Why did it pick Haiku over Opus?" resolves
directly from this record.
Prompts are fingerprints, not files
Prompts are never stored by default. Each request is fingerprinted with an HMAC-SHA256 keyed per tenant — one-way, exact-match only, and untestable without your key. The same prompt always produces the same fingerprint, so the audit log can answer how often an exact prompt arrived and how it was classified each time, without holding the prompt itself.
When you genuinely need to see what the classifier saw, debug capture is a time-boxed window during which prompts are stored encrypted, access-logged, and hard-deleted at expiry. Off by default, scoped to one API key or to low-confidence classifications, and refused entirely for tenants whose configuration promises zero retention.
The loop closes itself
You never report anything. The SDKs and gateway plugins observe what
production did with each pick — success, parse failure, escalation,
latency — and batch it back as fire-and-forget telemetry, keyed to the
same recommendation_id. Recommendation, decision trace, and
real-world result land in one ledger, per tenant, queryable, without a
single customer-side integration.
Telemetry is opt-out, and off means off — zero calls home. The only consequence is stated, never punished: without observed outcomes, your rankings stay purely benchmark-driven.
See it against your workloads.
Early access tenants get onboarding with real traffic, not a sandbox demo.
Request early access