Stop generating JSON just to pick a model: semantic routing with Gemini Decisions

In Smart Router, replacing our generative LLM classifier with Google's Gemini Decisions API (gemini-decision-1) made semantic routing faster, cheaper, and more accurate at the same time.

I built Smart Router for a plain economic reason. Most day to day prompts don't need a frontier model. Paying frontier prices and latency on routine lookups, formatting, or ticket triage adds up fast.

Our original baseline worked, at an awkward cost. On every request, the Go proxy built a separate rubric prompt ("You are a low-overhead API routing complexity classifier..."), wrapped the user's text inside it, and attached a JSON responseSchema. Then it called gemini-3.5-flash-lite just to generate {"complexity": "simple" | "medium" | "complex"}. When Google shipped gemini-decision-1 on November 11 while we were auditing the repo, that extra generation step looked backwards.

Why generative classifiers eat your routing budget

Let's be realistic about how teams usually build model routers today.

Option one is static character length and keyword rules. They cost nothing at the proxy layer, right until real user traffic arrives. A one line distributed systems bug looks 'short' and lands on your weakest model. A pasted email thread asking for a ZIP code looks 'long' and burns frontier compute.

Option two is the rubric prompt we were running. Asking a small generative model to write JSON understands intent, while charging a tax on every turn. You pay extra input tokens for the rubric. You wait on autoregressive decode latency while it prints JSON characters, and you pay for output tokens before your target model reads a word. Worse, a generated string gives you no calibrated confidence score. You can't tell an obvious lookup from a coin flip.

Routing isn't a writing task. It's a classification decision.

gemini-decision-1 scores structured choices in one forward pass without generating text (zero output tokens, priced as lightweight input scoring). In Smart Router, a single /v1beta/decisions call scores prompt complexity ('simple', 'medium', 'complex') and task kind ('coding', 'debugging', 'reasoning', 'summarization', 'triage') together. Each app can also pass its own plain English tier criteria.

The scores come back as calibrated probabilities across all three tiers. That lets you enforce a real MinConfidence floor. Uncertain calls escalate upward ('simple' to 'medium', or 'medium' to 'complex') instead of gambling on a weaker model.

Here is Smart Router's live Prompt Inspector running gemini-decision-1 on both ends of that length trap. First, a wordy shipping confirmation asking only for a ZIP code scores 'simple' and routes to gemini-3.5-flash-lite. Second, a one line consensus crash question ("Two-phase commit coordinator crashed after prepare. What blocks?") scores 'complex' and routes straight to gemini-4-argon:

Putting the three paths side by side

To compare the three approaches fairly, we ran all three live routing paths on Cloud Run against a 20-prompt adversarial suite built to fool naive routers.

The suite mixes terse concurrency and crash failures with zero code syntax, long operational notices hiding a single field lookup, and everyday retail questions stuffed with SQL / Go homographs ('select', 'class', 'package', 'return'). Smart Router also includes adapters for external and local rankers (Jev, Fastino, Contrastive-LM). We skipped those here because we didn't have live provider keys wired for them and didn't want to score offline stubs.

Side by side, the tradeoff is clear.

Character bounds missed almost every trap because prompt length pointed the wrong way. Our rubric-and-JSON prompt on gemini-3.5-flash-lite understood most of the suite, while adding three quarters of a second at the median just to decode JSON labels. gemini-decision-1 cut that routing latency by 3.5x with zero output tokens. Its single miss escalated an ambiguous homograph prompt upward to a safer tier rather than under-routing a hard question.

Testing those edge cases live also exposed three bugs in our own proxy before Friday's cut:

• Sticky multi-turn sessions (X-Session-ID) were ignoring mid-session X-Agent-Task upgrades when an agent shifted from triage into debugging.
• Our MinConfidence check was accidentally downgrading uncertain 'complex' prompts to 'medium' instead of keeping 'complex' as the ceiling.
• Our generative JSON baseline lacked a role field on Gemini platform calls, silently falling back to character heuristics until we inspected the response headers.

Trying dynamic routing on your own traffic

If your apps or agents still send every request to one default model, you're either overpaying on routine turns or under-powering the hard ones.

Trying dynamic routing doesn't require rewriting client code. Deploy Smart Router to Cloud Run (or run the Go binary locally with Application Default Credentials) and point existing Google GenAI or OpenAI SDK clients at gemini-dynamic. Want zero client config changes? Turn on Always Override on an app profile and let the gateway route behind the scenes.

In Smart Router's Complexity Routing view, setup takes three choices. Select Gemini Decisions API as the engine. Map Simple, Medium, and Complex to your target models (gemini-3.5-flash-lite, gemini-3.8-flash, and gemini-4-argon, or open weights Gemma 4 if you've configured a cluster), and set the confidence floor where uncertain prompts escalate upward:

Once real traffic flows through it, the impact shows up fast. Routine extractions, formatting, and triage peel off to Flash-Lite without hand maintained regexes. Hard reasoning lands on Argon automatically. Every response carries X-Routed-Model and X-Router-Confidence headers alongside the dashboard's live Prompt Inspector and Cost Analytics, so you can see why each turn landed where it did and refine your plain English tier criteria over time. Even without a full gateway, anywhere an agent workflow burns a generative LLM call just to pick a branch or tool can swap in one /v1beta/decisions call instead.

Next on our list in Smart Router is wiring live credentials for Jev, Fastino, and Contrastive-LM so we can compare dedicated external and embedding rankers against gemini-decision-1 on the same suite.

Are you routing across model tiers today, or still sending every request to one default model? Let me know in the comments!

$ go run ./cmd/smartrouter --config config.yaml
ROUTE DECISION: tier=complex (model=gemini-4-argon)
ENGINE: gemini-decision-1 · LATENCY: 212 ms · OUTPUT TOKENS: 0
CONFIDENCE: 0.94 (MinConfidence=0.70) · HEADERS: X-Routed-Model, X-Router-Confidence