LLM Routing and Cascade Inference: Cost-Quality Optimization at Serving Time

The economics of LLM serving create a brutal tradeoff: frontier models (GPT-4 class) cost 10-50x more per token than capable smaller models (GPT-3.5 class), yet most production queries don't require frontier-level reasoning. Studies consistently show that 60-80% of real-world API traffic can be handled by smaller models with no detectable quality loss. The question becomes: how do you automatically route each query to the cheapest model that can handle it?

This is the LLM routing problem, and 2024-2025 research has produced surprisingly effective solutions. The core insight: rather than building one model to rule them all, build a cheap classifier that predicts which model tier each query needs, then route accordingly.

The Cascade Architecture

The simplest routing strategy is a cascade: try the cheap model first, and only escalate to the expensive model if the cheap one is likely wrong.

Query → Small Model → Confidence Check → [High] → Return response
                                        → [Low]  → Large Model → Return response

FrugalGPT (Chen et al., 2023) formalized this as a sequential decision problem. Given a budget constraint and a set of models ordered by cost, the system learns a scoring function that decides at each stage whether the current response is "good enough" or needs escalation.

The scoring function is trained on a small labeled dataset:

class CascadeRouter:
    def __init__(self, models, scorer):
        self.models = models  # ordered by cost ascending
        self.scorer = scorer  # learned quality predictor

    def route(self, query, quality_threshold=0.85):
        for model in self.models:
            response = model.generate(query)
            score = self.scorer(query, response)
            if score >= quality_threshold:
                return response, model.cost
        return response, self.models[-1].cost  # fallback to best

The key challenge: the scorer must be much cheaper than the models it gates. FrugalGPT uses a DistilBERT-sized model (~67M params) trained on a few thousand (query, response, quality_label) triples. This adds ~1ms of latency per routing decision versus seconds for a frontier model call.

RouteLLM: Learned Preference Routers

RouteLLM (Ong et al., 2024) takes a different approach. Instead of scoring responses after generation, it predicts before generation whether the query needs a strong or weak model. This eliminates the latency of generating a response just to reject it.

The router is trained on preference data from Chatbot Arena, where human judges compared outputs from different models on identical prompts. The training signal: if humans can't distinguish the weak model's output from the strong model's output on a given query type, route to the weak model.

RouteLLM trains four router architectures and finds that a similarity-weighted ranking (SW) router works best:

class SWRankingRouter:
    """Routes based on similarity to training queries where
    the strong model was necessary."""
    
    def __init__(self, embeddings, labels, k=64):
        self.index = build_ann_index(embeddings)
        self.labels = labels  # 1 = needed strong model
        self.k = k

    def route(self, query_embedding, cost_threshold=0.5):
        neighbors, distances = self.index.search(query_embedding, self.k)
        similarities = 1.0 / (1.0 + distances)
        
        # Weighted vote: similar queries that needed strong model
        strong_weight = sum(
            sim * self.labels[idx] 
            for sim, idx in zip(similarities, neighbors)
        )
        total_weight = sum(similarities)
        
        strong_probability = strong_weight / total_weight
        return "strong" if strong_probability > cost_threshold else "weak"

The cost_threshold parameter gives operators a continuous dial between cost and quality. At threshold=0.5, RouteLLM achieves GPT-4 level quality on Chatbot Arena benchmarks while routing ~65% of queries to the cheaper model, cutting costs by roughly 2x without measurable quality degradation.

Matrix Factorization Routers

RouteLLM's most surprising finding: a simple matrix factorization router matches or beats the neural approaches. The intuition comes from recommendation systems — model the (query_type, model) interaction as a low-rank matrix, then predict which model will perform well for a new query.

Each query is embedded into a latent space, and each model gets a learned embedding vector. The routing score is their dot product:

class MFRouter(nn.Module):
    def __init__(self, embed_dim, hidden_dim, num_models):
        super().__init__()
        self.query_encoder = nn.Sequential(
            nn.Linear(embed_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim)
        )
        self.model_embeddings = nn.Embedding(num_models, hidden_dim)

    def forward(self, query_embed, model_idx):
        q = self.query_encoder(query_embed)
        m = self.model_embeddings(model_idx)
        return torch.sigmoid(torch.dot(q, m))

This router adds <0.5ms latency per query and requires only a few MB of parameters — negligible compared to even the smallest LLM.

Hybrid Serving with Speculative Routing

The most sophisticated systems combine routing with speculative execution. Rather than making a binary route-or-don't decision, they run both models in parallel and cancel the expensive one if the cheap model's confidence exceeds a dynamic threshold.

The economics work when the cheap model's cost is genuinely negligible relative to the expensive model. If the large model costs 30x more per token, you can speculatively run the small model on 100% of traffic with minimal cost overhead, then only invoke the large model for the ~30% of queries flagged as difficult.

async def hybrid_serve(query, router, small_model, large_model):
    # Launch cheap model immediately (always runs)
    small_task = asyncio.create_task(small_model.generate(query))
    
    # Router decides if we also need the large model
    route_score = router.predict(query)
    
    if route_score > THRESHOLD:
        # Hard query: launch large model, discard small result
        large_task = asyncio.create_task(large_model.generate(query))
        small_task.cancel()
        return await large_task
    else:
        # Easy query: use small model result
        return await small_task

In production, this pattern reduces p50 latency (most queries hit the fast path) while maintaining p99 quality (hard queries still get frontier-model treatment).

Training Routers Without Labels

A practical challenge: collecting human quality judgments is expensive. Recent work explores self-supervised routing using proxy signals:

Perplexity gap: If the small model's perplexity on a prompt is much higher than expected, route to the large model.
Consistency probing: Ask the small model the same question twice. High variance in responses suggests uncertainty, triggering escalation.
Feature-based: Certain query features (multi-step reasoning, code generation, multilingual) correlate with needing stronger models. A simple logistic regression on query features achieves 80%+ of the fully supervised router's performance.

Production Considerations

Calibration drift: Model capabilities change with updates. A router trained against GPT-4-0613 may misroute after GPT-4-turbo replaces it. Production systems need periodic recalibration — even a few hundred labeled samples per month suffices.

Multi-objective routing: Cost isn't the only dimension. Latency-sensitive paths (autocomplete, real-time chat) should prefer the fastest model that meets quality, while batch paths (summarization, analysis) can tolerate latency for cost savings.

Graceful degradation under load: When the expensive model is overloaded, increase the routing threshold dynamically — accept slightly lower quality rather than queueing. This creates natural backpressure:

def dynamic_threshold(base_threshold, queue_depth, max_queue):
    pressure = min(queue_depth / max_queue, 1.0)
    return base_threshold + (1.0 - base_threshold) * pressure * 0.5

Results in Practice

Deployed routing systems consistently report 2-4x cost reduction at equivalent quality, with careful tuning achieving up to 10x on narrow domains where the cheap model is highly capable (e.g., simple factual QA, code formatting, translation between common language pairs).

The field is converging on a design principle: model serving infrastructure should treat model selection as a first-class routing decision, not a static configuration. Just as CDNs route requests to the nearest edge server, LLM serving systems should route queries to the cheapest capable model — dynamically, per-request, with learned policies that adapt as models and traffic evolve.