Rent the Model, Own the Evals

As I write this, the #1 model on the Arena leaderboard is Claude Fable 5. In April 2025 the top spot belonged to Google's Gemini 2.5 Pro -- the first time Google held it. In August 2024 it was Anthropic's Claude 3.5 Sonnet, in a statistical tie with OpenAI's GPT-4o. One independent tracker counts 21 lead changes at the top since mid-2023; the leaderboard's own changelog shows new frontier models from Anthropic, OpenAI, Google, DeepSeek, Alibaba, and Moonshot arriving almost monthly, and the current top ten sits within about 20 Elo points -- inside the noise. The money has rotated just as hard as the rankings. In 2023, OpenAI had 50% of enterprise LLM API spend and Anthropic had 12%. Today Anthropic holds 40%, OpenAI 27%, and Google has tripled to 21%. Every one of those numbers will look different again next year.

I run a company that builds on these models, and I hold a view that sounds strange for someone who just published an essay about AI deployment: I do not particularly care which lab is winning this quarter. Claude is the best model for several of our workloads today. Six months ago the answer was different, and six months from now it may be different again. What I care about -- what I think any operator should care about -- is that when the answer changes, switching costs us days, not quarters.

That property does not come from the models. It comes from everything we built around them: the evaluation sets that define what "good" means for each of our workflows, the test harnesses that run every candidate model against those definitions, and the pipelines that treat the model as a component with a spec rather than a brand with a personality. The eval tooling company Braintrust states the principle plainly in its manifesto: the eval is as important as the model. I would go further, in procurement terms: the model is rented; the eval is owned.

A 300x price spread is a procurement problem

Back in April I did something with our AWS Bedrock console that I would do with any strategic input: I built a tiering sheet. Every available model, its list price per million tokens, its measured intelligence and coding scores, its speed, blended into a comparable cost figure the way Artificial Analysis does it -- three parts input to one part output. Two things jumped out.

First, the spread. The most capable model on the sheet, Claude Opus 4.7, blends to about $20 per million tokens. The cheapest usable one runs $0.04. That is a 300x range for what a purchasing department would call the same commodity category -- and it has widened since April. At this week's list prices, OpenAI's premium reasoning tier prices output at $180 per million tokens while DeepSeek's V4 Flash prices it at $0.28 -- roughly 640x -- and once cached input enters the math, the spread between the most and least expensive way to process the same token crosses 10,000x. Intelligence per dollar varies by a factor of 30 between models that are all, by any 2023 standard, astonishingly good.

Second, the shape. Price does not track capability linearly -- it tracks it in tiers, with a Pareto frontier running from open-weight workhorses up to the frontier models, and a scattering of dominated options you should simply never buy. The most instructive dot on the chart is the legacy trap: Claude Opus 4, still listed at $15/$75 per million tokens -- three times the price of its own successors for less capability -- for no reason other than that nobody repriced it and some buyers never rechecked. Every procurement person who has ever found a five-year-old part number still billing at launch price knows this dot.

To make it concrete, I priced four identical workloads -- ten million chatbot turns, a million RAG answers, half a million coding-agent turns, a hundred thousand document extractions -- across the tiers. Same work, every month:

Frontier models at the top of the sheet run that bundle at $252,500 a month. The bottom of the sheet does it for $3,280. Nobody should run all four workloads on either extreme -- that is the point. You do not need Opus or Fable to copy cells from one table to another or to add up a column of numbers -- and yes, I am aware Fable is #1 on the Arena this week; that is precisely the kind of thing that changes -- any more than you need aerospace-grade titanium for a shelf bracket. And you genuinely do not want the $0.04 model reasoning through a supplier contract. The question is never "what is cheapest"; it is "what is the cheapest model that still passes the eval for this task." Which is, word for word, a should-cost question.

Here is the sheet itself, abridged to the rows that tell the story:

* On the Pareto frontier. Blended $ per 1M tokens (3:1 input:output), AWS Bedrock list prices, April 17, 2026; Score = intelligence index, Artificial Analysis methodology. Download the full workbook (xlsx) -- all 68 models, the four-workload cost calculator, value rankings, and use-case recommendations.

There are three more lines from that spreadsheet exercise worth stealing. Reasoning models hide their true cost: they look cheap on the sticker but burn two to five times the output tokens in thinking traces, so a "cheaper" reasoning model can out-spend a pricier standard one per task. Caching and batch tiers change the math more than model choice: cache reads price at 90% off input or better, and batch runs at half price -- for agent workloads that re-read the same context, that is a bigger lever than switching vendors. And fast-cheap models are underrated on total cost, because what actually drives spend is cost per accepted outcome, not per-token price -- a model that finishes the job in one attempt beats a cheaper one that needs three.

One more force multiplies all of this: the whole price surface is collapsing as you negotiate on it. a16z measured what it calls LLMflation -- the cost of constant-capability inference falling about 10x per year -- and Epoch AI now estimates the cost of a fixed level of performance halves roughly every two months. A recent analysis of six years of token prices found a 600-fold decline since 2020, with a structural break in mid-2024 when the driver shifted from technology gains to open competition. In any other category, procurement would read that chart and refuse to sign a twelve-month fixed unit price. The same discipline applies here.

What procurement actually knows that IT is relearning

The discipline I am describing has a name in my industry: guided buying. Good procurement was never just getting the right price. It is guiding the organization to the right spec -- steering the engineer away from the aerospace alloy when the shelf bracket is fine -- and then monitoring usage so the policy survives contact with real people. Gartner has started saying the same thing to CIOs directly: its 2026 guidance on buying AI urges leaders to use sourcing and procurement as strategic advisors on business-led AI purchases, warns about the hidden costs beyond the API bill, and calls FinOps discipline critical to getting ROI from agents at all. Enterprise software has spent two years relearning this for tokens, mostly the hard way: hand out seats, watch the bill, wonder where the value went. I told the story of our head of recruiting's $500 token month in the deployment essay; the punchline there was pipeline design, but the quieter lesson was model tiering. (It is also the sequel to the order-of-operations argument: fluency first, then automation -- and then, apparently, a rate card.) His workload never needed a frontier model. Nobody had told him, because nobody owned telling him.

The market has already voted for the multi-model version of this. In a16z's latest enterprise survey, 81% of CIOs report using three or more model families, up from 68% a year earlier -- "the multi-model world is here to stay" is a16z's own phrasing -- and 37% of enterprises run five or more models in production. Menlo Ventures finds 60% routing prompts to whichever model performs best for the task. The gateway layer has industrialized to match: LiteLLM fronts a hundred-plus models behind one API, AWS ships reference architecture for multi-provider gateways, and OpenRouter's request-level dataset -- 100 trillion tokens of it, analyzed with a16z -- shows substitution happening in real time. In June, AWS even productized the switch itself: its Transform service now runs model-to-model migration assessments, moving workloads off OpenAI or Gemini or a direct Anthropic SDK while preserving the application around them. Switching is becoming a workflow category with tooling, which tells you which way the market thinks the pressure runs.

The LangChain agent-engineering survey has a detail I find more revealing than any of those numbers: 89% of teams have observability on their AI systems, but only 52% have evals. Everyone can see what their models are doing; barely half can say whether it is good. That gap -- watching without judging -- is exactly the gap between having a spend dashboard and having a sourcing strategy.

Closing it looks like ordinary procurement mechanics applied to a weird new commodity:

The monitoring row deserves one emphasis: the point of the token-budget alert is the conversation that follows -- what are you trying to do, and is this the right tool for it? -- not the ceiling itself. That conversation is where the tiering policy survives contact with enthusiastic people.

The evals are the asset

All of this depends on one piece of infrastructure, and it is the least glamorous one: the evaluation set. A real eval is not a benchmark score copied from a launch post -- OpenAI's own enterprise guidance concedes that frontier benchmarks "cannot reveal all the nuances" of a specific workflow in a specific business. A real eval is a few hundred input-output pairs sampled from your actual work -- this supplier email, this contract clause, this candidate profile -- with a definition of "good" that your own operators wrote. Golden datasets, the practitioners call them. Anthropic's engineering guidance goes a step further and recommends writing the evals before the capability works, eval-driven development, the way disciplined teams write tests before code. They are tedious to build, they require your best people's judgment, and they are the single most durable thing in your AI stack.

Durable because everything else rotates. The model changes -- the top of the leaderboard has passed between three labs in two years. The prompts change with the model, because prompts are couplings, not assets; anyone who has swapped models under a production workload knows the prompt that sang on one model goes flat on another. Even the leaderboard changed its own name twice. The eval set is the one thing that gets more valuable with every rotation, because every new model release turns into a same-day answer to a concrete question: does this pass our bar, at what price, on our work? In practice the machinery is simple to describe: a stable task interface, an adapter layer so any model can sit behind it, an eval gate that scores correctness, cost, latency, and safety, a canary that routes a sliver of real traffic to the candidate, and a promotion rule -- the new model ships only when it clears the bar at a better cost per accepted outcome. Teams with that machinery experience a model launch as a shopping opportunity. Teams without it experience a migration project.

This is also the honest answer to the strongest counterargument, which deserves stating properly: switching is never actually free. Prompt behavior couples to models. Caching discounts and volume pricing reward concentration. Agent reliability compounds with familiarity, and one recent measurement study makes the deeper point that what you actually buy is not "a model" but a provider-specific endpoint -- same weights, different latency, throughput, reliability, and effective price depending on who serves them. Governance teams reasonably prefer one audited vendor and one throat to choke. And evals themselves are not free: they go stale, and a suite that passes offline can still miss a production failure -- Anthropic's own guidance warns that no single evaluation layer catches everything. The subtlest failure is judge drift: if a model grades your outputs, upgrading the judge silently rewrites the rubric, so serious eval systems pin judge versions and re-calibrate against humans on a schedule. All true. But notice that every one of those switching costs shrinks as eval coverage grows, and none of them reverses the direction of the argument: standardizing on one vendor may be the right call this year, and it is still cheaper to hold that position with an exit built than without one. Models regress, endpoints get deprecated, a safety update changes refusal behavior -- a model that passed in March can fail a production edge case in July, and evals with a canary route are how you see it before your users or your CFO do. The negotiation math alone can pay for the harness: a vendor who knows that leaving means six months of prompt rewrites has a ceiling on your discount; a vendor who knows 20% of your traffic can move to an eval-certified alternate within two weeks does not. You do not build model flexibility by refusing to commit; you build it by making re-commitment cheap. Optionality is not a posture. It is a capability, and it is purchased in advance.

Open weights, on-prem, and the sovereignty turn

The flexibility argument extends below the API line, because 2026 is the year self-hosting stopped being a hobby. On my April sheet, the best open-weight models -- DeepSeek, Qwen, MiniMax -- sit on the same Pareto frontier as the commercial workhorses; on today's leaderboard, MIT-licensed DeepSeek variants and Apache-licensed Gemma sit inside the frontier cluster, and DeepSeek's own V3.2 technical report claims parity with the top commercial reasoning models. The serving stack underneath (vLLM, NVIDIA NIM, SGLang, TensorRT-LLM) is production-hardened, and the hardware vendors have built the sales motion: HPE is selling private-cloud AI explicitly on token-cost control and sovereignty, and Dell claims its on-prem agentic stack cuts spend up to 87% versus cloud APIs over two years -- vendor math, but directionally the same argument as owning any capacity you run hot.

The demand side has turned too, and not primarily for cost reasons. In a March survey of enterprise AI infrastructure, 79% of organizations said they had moved at least some AI workloads off the public cloud in the past two years, and 91% of IT decision-makers said they would choose on-prem, private, or hybrid infrastructure for AI touching sensitive data. Gartner coined a word for the geopolitical version -- geopatriation -- and projects that by 2030 three-quarters of European and Middle Eastern enterprises will relocate workloads for sovereignty reasons. Data residency, enforceable AI regulation, and the general realization that inference on your customers' data is a supply chain, not a subscription, are doing to AI what they long ago did to manufacturing footprints.

Two honesty notes belong here. First, the spend data cuts the other way: open-source models' share of enterprise LLM API spend has actually fallen, to around 11-13%, because supported commercial models are easier to buy, secure, and blame. Both facts are true at once -- dollars concentrate in commercial APIs while sensitive and high-volume workloads migrate to controlled infrastructure -- and the reconciliation is exactly the tiering logic again: the mix is the strategy. Second, self-hosting moves the model line item from opex to a capacity commitment, and I have written about what capacity commitments do to you when demand shifts. Owning the GPUs is the same bet an automaker makes owning a stamping line: cheaper per unit, exactly as flexible as your forecast -- the classic make-versus-buy decision, now with a power bill.

Frontier models or tokenmaxxing?

The most interesting live question in model economics is whether you buy intelligence at the top of the price sheet or manufacture it at the bottom. The evidence that you can manufacture it is real. The Stanford "Large Language Monkeys" work showed that repeated sampling scales like a production process: on SWE-bench Lite, a mid-tier open coding model solved 15.9% of issues in one attempt -- and 56% when allowed 250 attempts, beating the 43% single-shot state of the art from frontier models at under a third of the price. Coverage grows log-linearly with samples, and DeepMind's test-time-compute work sharpened the point: allocating inference compute adaptively can be more than four times as efficient as naive best-of-N. Intelligence, in the aggregate, is purchasable in volume from cheaper suppliers.

The catch is the verifier. Repeated sampling works when you can check answers automatically -- unit tests, schema validation, reconciliation against a system of record. Where verification is weak, best-of-N decays fast; the formal results on imperfect verifiers confirm what practitioners feel, which is that a mediocre judge squanders a good ensemble. And the reasoning-token effect works against naive tokenmaxxing too: thinking traces quietly multiply output spend, which is how a "cheap" reasoning model produces an expensive invoice.

So the practical answer for an operating company, rather than a lab, is boring and useful. Where your task has a built-in verifier -- code with tests, extraction with schemas, math with reconciliation -- cheap models sampled generously are the best intelligence per dollar on the market, and getting better every quarter. Where the task is judgment -- an agent running unsupervised, a customer-facing answer, a decision with money attached -- cost per accepted outcome dominates unit price, and the frontier model is usually the cheap option once you count retries, escalations, and trust. The frontier buys reliability per attempt; the volume tier buys attempts. Guided buying, again: match the spec to the job, and let the evals referee.

The sourcing team you already have

If this whole essay reads like a procurement playbook wearing an AI costume, that is the point. A volatile supplier landscape where the leader rotates every few quarters; a several-hundred-x price spread inside one category; unit prices halving every couple of months while volumes explode; spec-matching that most requesters cannot do themselves; a build-versus-buy decision with real capacity risk on the build side. Procurement teams have run exactly this movie for decades on multi-sourced commodities -- qualify more than one supplier, define acceptance tests, tier the catalog, monitor usage, and never let a single vendor own your spec. The companies treating tokens this way are not being clever. They are being ordinary, in the best way, about an input their software teams were treating as a religion.

That conviction is built into how we work at LightSource. Inside the product, model choice is a routing decision behind our own eval gates -- bid normalization, spec extraction, and cost analysis each run on whatever model currently clears our quality bar at the best cost per accepted outcome, some steps run on deterministic code because no model is the right spec, and the riskiest calls get a human, and our customers never need to know or care which lab that is this quarter. And for the sourcing teams we serve, the discipline is the same one they already apply to resins and castings, pointed at a new line item: total cost of ownership thinking for the most volatile commodity their company now buys.

The models will keep leapfrogging each other -- that is the happy outcome, the one where competition keeps working and the deflation keeps compounding. The uncomfortable version of the question is not which lab wins. It is whether, eighteen months from now, your organization will be able to take advantage of whoever wins -- or whether it will still be paying the legacy price on the dot at the far right of the chart, running everything through the model it happened to marry in 2025. The difference between those two companies is not taste in vendors. It is a few hundred rows of golden data and the discipline to route through them. Rent the model. Own the evals.

Sources

• Arena (LMArena) Text leaderboard and leaderboard changelog -- Claude Fable 5 at #1 (July 10, 2026, 7.27M votes, 374 models); near-monthly frontier additions; BenchLM tracker counts 21 lead changes since 2023.

• LMSYS style-control analysis (Aug 2024) -- Claude 3.5 Sonnet tied #1 with GPT-4o; Gemini 2.5 Pro takes #1 (Apr 2025).

• Menlo Ventures -- Mid-year LLM market update and TechCrunch coverage -- enterprise API spend shares (Anthropic 40%, OpenAI 27%, Google 21%); 37% on 5+ models; open-source share ~11-13%.

• a16z -- Welcome to LLMflation and enterprise AI survey -- ~10x/year deflation; 81% of CIOs on 3+ model families; "the multi-model world is here to stay."

• Epoch AI -- inference cost trends -- cost of fixed capability halving roughly every 2 months; arXiv 2603.28576 -- ~600x token-price decline 2020-2026 with a mid-2024 structural break.

• Spencer Penn's AWS Bedrock model tiering analysis (April 17, 2026) -- 300x blended-price spread, Pareto frontier, workload cost model; intelligence scores per Artificial Analysis methodology. Current-week anchors: GPT-5.5 pricing, Claude Opus 4.8, Gemini 3 Pro, DeepSeek V4 pricing.

• Braintrust -- Eval Manifesto -- "the eval is as important as the model"; OpenAI -- Evals drive the next chapter of AI -- frontier benchmarks "cannot reveal all the nuances"; Anthropic -- Demystifying evals for AI agents -- eval-driven development and its limits.

• AWS Transform -- model-to-model migration assessments (June 2026) -- switching productized; LiteLLM and AWS multi-provider gateway guidance; OpenRouter data -- 100T-token analysis with a16z.

• LangChain -- State of Agent Engineering -- 89% observability vs 52% evals.

• Large Language Monkeys -- 15.9% -> 56% at 250 samples vs 43% frontier single-shot at <1/3 cost; DeepMind/Berkeley test-time compute -- compute-optimal scaling >4x more efficient than best-of-N; ROC-n-Reroll -- verifier imperfection.

• Gartner -- Buying AI: a CIO's guide, FinOps for AI agents, and the AI cost explosion note.

• Cloudian -- AI Infrastructure Survey (March 2026) -- 79% moved AI workloads off public cloud; 91% prefer controlled infra for sensitive data; HPE private cloud AI and Dell on on-prem agent economics; DeepSeek V3.2 technical report.

• Provider-specific endpoint measurement study -- you buy an endpoint, not a model; routing counterfactuals.

Frequently Asked Questions

Why does model flexibility matter for enterprise AI?

Because leadership rotates constantly: the top of the main industry leaderboard has passed between Anthropic, OpenAI, and Google repeatedly in two years, and enterprise spend share has swung just as dramatically (Anthropic from 12% to 40%, OpenAI from 50% to 27%). Organizations architected around a single vendor experience each rotation as a migration project; organizations with evals and routing experience it as a pricing opportunity. Flexibility also captures the deflation -- the same capability gets roughly 10x cheaper every year.

What are LLM evals and why are they the durable asset?

An eval is a test suite for AI work: a few hundred real input-output pairs from your own workflows, with acceptance criteria your operators defined. Unlike prompts (which couple to specific models) and model choices (which rotate), eval sets appreciate over time -- every new model release can be scored against them within a day. OpenAI's enterprise guidance notes that public benchmarks cannot capture a specific business's nuances, which is exactly why owned evals, not leaderboard positions, should gate model decisions.

How big are the price differences between LLMs?

On AWS Bedrock in April 2026, blended prices spanned roughly 300x; at July 2026 list prices the extremes are wider still -- premium reasoning output at $180 per million tokens versus $0.28 on an ultra-cheap tier, with cached-input spreads crossing 10,000x. Priced against identical monthly workloads, the same work costs $252,500 on a frontier model and $3,280 at the budget tier. The practical question is the cheapest model that still passes your eval for each task.

Is it cheaper to use many calls to a cheap model than one call to a frontier model?

For tasks with automatic verification -- code with unit tests, extraction with schema checks -- yes, often dramatically: repeated-sampling research showed a mid-tier model solving 56% of SWE-bench issues with 250 attempts versus 43% for single-attempt frontier models, at under a third of the cost, and compute-optimal test-time scaling improves on that by 4x or more. Without a reliable verifier the advantage decays quickly, and reasoning models' hidden thinking tokens can erase apparent savings. For judgment-heavy or customer-facing work, frontier models usually win on cost per accepted outcome.

Should companies self-host open-weight models?

For high-volume, verifiable, or data-sensitive workloads, increasingly yes: open-weight models now sit inside the frontier quality cluster, serving infrastructure is production-grade, and 79% of enterprises have already moved some AI workloads off public cloud -- with Gartner projecting large-scale "geopatriation" for sovereignty by 2030. But open-source models' share of API spend has actually fallen, because supported commercial APIs remain easier to buy and operate. Most organizations land on a mix -- commercial frontier models for judgment, self-hosted open weights for volume and sovereignty -- which is a portfolio decision, not an ideology.

What is guided buying for AI models?

Guided buying is the procurement practice of steering requesters to the right specification, not just the right price -- and monitoring usage so the policy holds. Applied to AI, per Gartner's 2026 guidance on AI cost control: maintain a tiered model catalog, route simple tasks to cheap tiers by default, require justification for frontier usage, track cost per accepted outcome by workflow, and refresh the catalog quarterly as prices deflate. It treats models as a managed commodity category rather than a badge of loyalty.