LLM Selection: Benchmarking, Routing, and Fallbacks

Public leaderboards measure general ability. Your decision depends on your tasks, latency budget, and cost. Build a private benchmark, then route requests so each uses the cheapest model that passes.

Build a task benchmark

Take 100 or more real inputs per task type with graded expectations. Score quality, latency percentiles, and cost per request for each candidate model.

def benchmark(models, cases, grader, price):
    rows = []
    for name, call in models.items():
        scores, lat, cost = [], [], 0.0
        for c in cases:
            t0 = time.monotonic()
            out = call(c["input"])
            lat.append(time.monotonic() - t0)
            scores.append(grader(c, out.text))
            cost += out.in_tokens * price[name]["in"] / 1e6 + out.out_tokens * price[name]["out"] / 1e6
        lat.sort()
        rows.append({"model": name, "quality": sum(scores) / len(scores),
                     "p95_s": lat[int(0.95 * len(lat)) - 1], "cost": cost})
    return rows

Choose tiers

Typically two or three tiers suffice: a small model for simple, high-volume tasks, a large model for hard reasoning, and optionally a specialist for code or extraction. Pick the cheapest model that meets the quality threshold for each task.

Route

Route by task type first, which is deterministic and easy to reason about. Add confidence-based escalation: try the small model, and send to the large one if a validator or confidence check fails.

def answer(task, payload, gateway, validate):
    first = gateway.complete(task, "small", build(task, payload))
    if validate(first.text):
        return first
    return gateway.complete(task, "large", build(task, payload))      # escalate

Fallbacks

Providers have outages and rate limits. Define an ordered list of equivalent models per tier and fail over automatically, after testing that prompts behave acceptably on each.

Keep it swappable

• Isolate provider SDKs behind the gateway.
• Store prompts as versioned templates.
• Rerun the benchmark when a new model arrives or prices change.
• Pin model versions in production and upgrade deliberately.

Data residency and privacy

Some tasks must stay on private infrastructure. Include self-hosted models as a tier, and evaluate them with the same benchmark.

Reading benchmark results

A useful benchmark report is a table with one row per model and columns for quality, latency, and cost, broken down by task category. Look for the cheapest model that clears your quality threshold in each category, and look at the failures of the cheaper models. Sometimes a model fails only on a narrow input type that a simple rule can route elsewhere.

Confidence-based escalation, carefully

Escalation depends on a reliable signal. Validators, such as schema checks and arithmetic, are strong signals. A model's own stated confidence is weak. Evaluate the escalation rule on your benchmark: measure how many requests escalate, and how much quality improves, to confirm that the saved cost is real.

def escalation_report(cases, small, large, validate, grader):
    esc, quality = 0, []
    for c in cases:
        a = small(c["input"])
        if validate(a):
            quality.append(grader(c, a))
        else:
            esc += 1
            quality.append(grader(c, large(c["input"])))
    return {"escalation_rate": esc / len(cases), "quality": sum(quality) / len(quality)}

Operational concerns

• Version pinning: pin exact model versions and test before upgrading.
• Deprecations: track provider announcements and schedule migrations.
• Rate limits: design for provider quotas, and spread load across providers when contracts allow.
• Prompt portability: keep prompts per model family, because wording that works for one may not transfer.

Self-hosting

Self-hosted open-weight models give control over data and predictable costs at high volume, and they bring operations work: hardware, scaling, upgrades, and security patching. Compare the full cost with managed options at your expected volume, and include the staff time to run them.

Re-benchmarking cadence

New models and price changes arrive often. Re-run the benchmark quarterly and whenever a candidate is announced that targets your workload. Because the harness already exists, the cost of checking is small, and the potential savings or quality gains can be significant.

How we can help

We build your private benchmark and a gateway with routing and failover, and we hand over the harness so you can re-evaluate when the market moves. Ask for a model selection sprint.