Manu Bhardwaj · Papers

The Heterogeneous-GPU Margin. Coral and the Multi-LLM Procurement Problem.

A Daily Field Note on Heterogeneous Hardware and Joint Multi-Model Allocation

Manu Bhardwaj. ifitsmanu.com. 11 May 2026. Last updated 11 May 2026. Version 1.0. Field Notes #6.

Cite this article. Research index. Companion. The Inference Stack in 2026.

Daily field note. Second piece in the daily-review cadence. One fresh paper or post in the inference-economics or verification-economics wedge from the previous seven days, decomposed against what it shows and what it does not. Lighter scope than Field Notes #1 to #4. Same voice.

What it claims

Mei, Li, Chen, Pan, Wu, Miao, Jia, and Rashmi (2026), posted to arXiv on 5 May 2026, introduce Coral. An “adaptive heterogeneity-aware multi-LLM serving system” whose stated economic claim has two parts. First, the LLM market is now fragmented across many models with no dominant winner, so a serving fleet typically hosts several models concurrently. Second, cloud GPU supply is now heterogeneous, with mid-tier and older-generation GPUs delivering comparable performance per dollar to top-tier hardware while enjoying better availability. Coral jointly optimizes the resource allocation and serving strategy of each model replica across all models in the fleet, rather than tuning each model replica in isolation.

The headline numbers. Across 6 models and 20 GPU configurations, Coral reduces serving cost by up to 2.79x over the best baseline and delivers up to 2.39x higher goodput under scarce resource availability. The two-stage decomposition that makes the joint problem tractable cuts online solve time from hours to tens of seconds while preserving joint optimality.

How it argues

Coral’s central observation is that the per-replica optimization problem is solved for a single model on a fixed GPU configuration, while the procurement problem the operator actually faces is joint. Which models go on which GPUs, how many replicas of each, and what serving strategy each replica uses, all under one cost-and-availability constraint.

The structural argument is that this joint problem is not decomposable into per-model subproblems without losing optimality, because the GPU pool is shared. A replica of model A on an H100 forecloses that H100 for model B; a replica of model A on an A100 cluster trades off against model C’s latency budget on the same cluster. Coral preserves joint optimality through a two-stage decomposition that splits the search into a configuration-selection stage and an allocation stage, with a tractable coupling between them. The result is that the online solve drops from hours to tens of seconds, which is the threshold below which the optimizer can react to shifting demand and shifting GPU availability.

The empirical setup spans 6 models and 20 GPU configurations. That is the right shape of grid for the argument. Six models is more than the two-or-three regime where a hand-tuned schedule wins; twenty GPU configurations is broad enough to make the heterogeneity claim non-vacuous.

What is interesting

The interesting structural property is the move from per-replica tuning to fleet-level procurement as the binding cost lever.

In Field Notes #1 I argued that the four stack-level changes (quantization, runtime, decoding-time parallelism, hardware) compound but not linearly, and that the bottleneck changes as each improvement lands. Coral identifies a fifth lever that is structurally upstream of those four. The choice of which model lands on which GPU at all. Quantization, PagedAttention, continuous batching, and speculation all act inside the per-replica box. Coral acts on the assignment from models to boxes.

A second observation. The “comparable performance per dollar” claim for mid-tier and older-generation GPUs is the part that does the most economic work. If top-tier hardware dominated on every metric simultaneously, joint allocation would collapse to “buy more H100s.” The fact that mid-tier GPUs deliver competitive perf-per-dollar means the right schedule mixes hardware generations, and the value of joint optimization is the gap between the mixed schedule and the homogeneous one. The 2.79x cost reduction is the size of that gap on Coral’s grid.

A third observation. The 2.39x goodput improvement under scarce resource availability is the more interesting half of the result. Cost reduction at fixed availability is a steady-state claim. Goodput under scarcity is a dynamic claim, and scarcity is the regime that defines the procurement problem. Operators do not buy GPUs at list price from an infinite pool; they buy what is available, at the price that is available, often through spot tiers and reservations with shifting capacity. A joint allocator that retains goodput under shifting supply is doing the work the procurement function actually needs.

What is missing

The paper reports cost and goodput lifts but several quantities that would make the framework directly comparable to the cost-economics framing are not surfaced.

First. No explicit decomposition of the 2.79x lift across the joint axes. How much of the gain comes from the hardware-mix axis (mid-tier plus top-tier versus top-tier only), how much from the model-coresidency axis (which models share a GPU class), and how much from the per-replica serving-strategy axis is left implicit. Knowing the decomposition would tell an operator whether the lever is procurement, packing, or runtime tuning, and the levers do not have the same operating cost.

Second. The model count is six. The fragmentation argument is strongest at higher model counts, where the joint-allocation problem branches combinatorially and the per-model heuristic loses the most. Whether the lift extends, holds, or contracts at fleet sizes of 20 to 50 hosted models is the binding question for hyperscaler deployment, and the paper does not answer it.

Third. The cost model is a serving cost, not a Verified Capability per Dollar (VCpD) number in the sense of Field Notes #1. Coral optimizes goodput against SLOs, which is closer to delivered-tokens-per-dollar than to verified-correct-tokens-per-dollar. The verifier-economics layer that Field Notes #2 and Field Notes #3 develop sits outside Coral’s loss function. A multi-LLM fleet that delivers high goodput but mixed verifier-pass quality across its models is not yet a cost-correct optimum. The integration of Coral-style joint allocation with α-aware routing is the open structural question.

Why it matters now

Two reasons. The first is a market-structure reason. The fragmentation premise is empirically correct as of May 2026. No single proprietary model captures the routing layer; production fleets host frontier and open-weight families concurrently, often through router and gateway abstractions. Joint allocation over heterogeneous GPUs is no longer a research curiosity; it is the operating problem for any team running more than one model at scale.

The second is a hardware-supply reason. The mid-tier-availability argument tracks the supply curve. H200 and B200 capacity remains rationed; A100, L40S, and prior-generation Hopper SKUs sit on cloud price sheets at delivered-token cost points that are competitive once joint allocation absorbs the heterogeneity. Operators who refuse to mix generations leave the 2.79x on the table because they are running an allocator that cannot use the cheaper supply.

The cleaner the procurement layer gets, the more of the cost surface it touches. Coral identifies the layer above the per-replica box and the layer below the verifier as the place where the next 2x in inference-economics lives. That is the lever this paper extends and the structural argument it adds to the inference-stack framework.

Source

Cite this article

@misc{bhardwaj2026heterogeneousprocurement,
  author = {Bhardwaj, Manu},
  title  = {The Heterogeneous-GPU Margin: Coral and the Multi-LLM Procurement Problem},
  year   = {2026},
  month  = {May},
  url    = {https://ifitsmanu.com/papers/heterogeneous-procurement},
  note   = {Field note. Field Notes \#6. Daily review of arXiv:2605.04357. Version 1.0.}
}

Research index. Home.