Python

SmartRouter: send each prompt to the cheapest model that can handle it

Not every prompt needs the biggest model. SmartRouter runs simple or private requests on a model on your own machine and sends the hard ones to Gemini in the cloud.

Architecture diagram of smartrouter policy-driven hybrid AI router dispatching prompts between Google Cloud Gemini platform (Gemini 3.8 Flash) and local Gemma 4 runners. — SmartRouter picks between a local Gemma 4 model and the Gemini platform for each prompt.

What it does

SmartRouter in plain terms

SmartRouter sits in front of your models and decides, one prompt at a time, whether to run it locally on Gemma 4 or send it to Gemini on Google Cloud. It scores how complex the request is, checks the token budget and any data-residency rule, and picks a route that fits your latency and cost limits. It records why it chose each route.

How it works

Three things to know

No API keys: it signs in to Google Cloud with your existing Application Default Credentials (ADC).
A fast classifier scores each request (code depth, tool calls, latency budget) in about a millisecond.
Every response carries a header saying which model ran and why.

Sample result: 410 ms Gemini route in a sample run.

Learn more

Essays and source

Read the essay: WIBTD/W/M: Routing AI responses between local runners → Related: What I'm building this week: Route your Gemini API calls on your terms → Related: WIBTD/W/M: A fool's folly with local AI models → GitHub: SmartRouter ↗

View a code sample (hybrid_model_router.py) Python

File: hybrid_model_router.py

from dataclasses import dataclass
from typing import Literal


@dataclass(frozen=True)
class RoutingPolicy:
    project_id: str = "davenport-boutique"
    location: str = "us-central1"
    frontier_model: str = "gemini-3.8-flash"
    fast_cloud_model: str = "gemini-3.5-flash-lite"
    local_edge_model: str = "gemma-4-9b-it"
    complexity_threshold: float = 0.65


def select_execution_route(
    prompt_tokens: int,
    requires_tool_calling: bool,
    complexity_score: float,
    policy: RoutingPolicy = RoutingPolicy(),
) -> dict[str, str | float | Literal["cloud-frontier", "cloud-fast", "edge-gemma"]]:
    """Select optimal Gemini 3.8+ or Gemma 4 target using explicit SLO rules."""
    if requires_tool_calling or complexity_score >= policy.complexity_threshold or prompt_tokens > 8192:
        return {
            "tier": "cloud-frontier",
            "model": policy.frontier_model,
            "project": policy.project_id,
            "location": policy.location,
            "auth": "gcloud-adc",
            "rationale": f"High complexity ({complexity_score:.2f}) or tool synthesis required.",
        }
    if prompt_tokens > 2048:
        return {
            "tier": "cloud-fast",
            "model": policy.fast_cloud_model,
            "project": policy.project_id,
            "location": policy.location,
            "auth": "gcloud-adc",
            "rationale": "Medium context window routed to Gemini 3.5 Flash-Lite.",
        }
    return {
        "tier": "edge-gemma",
        "model": policy.local_edge_model,
        "project": policy.project_id,
        "location": "local-runner",
        "auth": "local-ipc",
        "rationale": "Compact classification task eligible for Gemma 4 runner.",
    }

Run it: $ uv run python -m smartrouter.cli --prompt 'Refactor AST graph lock lease expiry' --slo-ms 850

ROUTE DECISION: vertex-ai/gemini-3.8-flash (project=davenport-boutique, location=us-central1)
REASON: multi-symbol code mutation (complexity=0.82 > threshold=0.65)
AUTH MODE: google-cloud-adc (zero API key) · EST. LATENCY: 410 ms