\documentclass[11pt]{article}

\usepackage{../../template/ifitsmanu-paper}

%% --- Paper metadata ---
\title{The Routing Premium.\\
       An Economic Threshold for Difficulty-Conditional Inference Compute.}
\author{Manu Bhardwaj}
\date{May 2026}

\renewcommand{\shortauthor}{Bhardwaj}
\renewcommand{\shorttitle}{The Routing Premium}

\begin{document}

\maketitle

\begin{abstract}
When does conditioning inference compute on a noisy estimate of task
difficulty reduce cost-per-correct-answer relative to a fixed-compute
baseline? Five published patterns route compute on a difficulty signal.
Two operate at the per-token or per-layer level: speculative decoding
\citep{leviathan2023speculative, cai2024medusa} and early-exit decoding
\citep{schuster2022calm}. Three operate at the per-query level: cascade
routing \citep{chen2023frugalgpt}, adaptive self-consistency
\citep{petullo2026veccisc}, and complexity-aware exploration
\citep{petullo2026casql}. None derives the threshold above which the
routing rule pays. We derive one. Under the \emph{Cost-correct}
decomposition of \citet{bhardwaj2026costcorrect}, the routing premium is
positive iff $\kappa \cdot \Delta > \gamma$ at the margin around the
unconditional optimum, where $\kappa$ is classifier calibration, $\Delta$
is workload heterogeneity in compute, and $\gamma$ is classifier
overhead. The condition unifies the five patterns as one allocation
rule. We calibrate against six published systems spanning all five
classes and find that every operating point sits on the positive side of
the threshold. The elasticity reading isolates which operating points
are close enough to fail under modest disclosure error.
\end{abstract}

% =========================================================================
\section{Introduction}
\label{sec:intro}

Adaptive inference is now a standard pattern. Speculative decoding
routes tokens between a drafter and a verifier
\citep{leviathan2023speculative, chen2023drafter, cai2024medusa,
li2024eagle}. Cascade systems route prompts across model tiers
\citep{chen2023frugalgpt, zhan2026mpd2}. Self-consistency systems prune
candidate traces by semantic similarity \citep{petullo2026veccisc}.
Tree-search systems vary exploration breadth by estimated complexity
\citep{petullo2026casql}. Early-exit decoders stop at confident layers
\citep{schuster2022calm}. All five report cost reductions at
iso-accuracy. None reports the threshold under which the routing rule is
rational.

The five literatures developed in parallel and have not been unified. A
practitioner reading them in isolation sees five engineering tricks.
Read together, they are five instances of one allocation rule: spend
more compute on tasks the system thinks are hard, less on tasks it
thinks are easy. The unifying object is a \emph{difficulty-conditional
compute policy}. The economic question is whether the difficulty
classifier earns its cost.

We frame the question as a population-level allocation problem. A
provider faces a workload distribution $F(d)$ over latent difficulties
$d \in [0, 1]$. The provider can run a single fixed-compute policy at
the unconditional optimum, or it can run a calibrated classifier and
route each query to a difficulty-specific compute level. The router
pays a classifier overhead. The question is when the savings from
re-allocating compute across the workload exceed the classifier
overhead at the margin.

The contribution is a closed-form threshold under Cost-correct. Routing
pays iff
\begin{equation}
\kappa \cdot \Delta \;>\; \gamma,
\label{eq:threshold}
\end{equation}
where $\kappa$ is the explained-variance calibration quality of the
classifier on the workload, $\Delta$ is a dimensionless measure of
workload heterogeneity built from the second derivative of
cost-per-correct-answer in compute, and $\gamma$ is the classifier
overhead as a fraction of the unconditional inference cost. The
threshold is local, holding to second order in the deviation $c(\hat d)
- \bar c^*$ around the unconditional optimum $\bar c^*$. The five
published patterns operate inside this local regime; we flag the
cascade specialization where the local-margin assumption binds hardest
and the third-order correction is non-trivial.

The contribution is non-trivial under the Cost-correct frame. The
unconditional optimum $\bar c^*$ already minimizes expected cost; the
routing premium is the \emph{second-order} gain from re-allocating
compute around that optimum, weighted by classifier calibration on the
workload, net of classifier overhead. The threshold is a statement
about a curvature-by-variance product, not a first-order gradient. It
is orthogonal to the channel-allocation threshold of Paper~\#1 of this
wedge \citep{bhardwaj2026inferencefrontier}, which fixes the
\emph{channel} (training versus inference) at a single representative
query; the present threshold sets the \emph{distribution within the
inference channel} at a heterogeneous query mix. The two thresholds
compose multiplicatively in production cost and
\Cref{sec:discussion} sketches the combined diagram.

We calibrate \eqref{eq:threshold} against six published systems spanning
all five allocation-rule classes. \Cref{tab:calibration} in
\Cref{sec:experiments} reports the system-by-class mapping and the
six-row calibration. Every operating point sits on the positive side of
the threshold; CALM is the smallest margin and the natural sensitivity
case. \Cref{sec:discussion} turns the result into
three implications for serving infrastructure design and identifies the
disclosure gap that would let independent researchers falsify
\eqref{eq:threshold}.

% =========================================================================
\section{Related work}
\label{sec:related}

Four bodies of literature bear on the question. None derives the
threshold. A fifth (the Cost-correct frame) supplies the cost
decomposition the threshold leans on.

\textbf{Speculative decoding.} \citet{leviathan2023speculative} and
\citet{chen2023drafter} derive the expected speedup of a
drafter-verifier scheme as a function of drafter accept rate and
relative drafter cost. The derivation is per-token and within-sequence:
the routing decision is the next-token verifier check, not a per-query
allocation. \citet{cai2024medusa} and \citet{li2024eagle} tighten the
drafter side with multi-head and tree-attention drafting and report
higher accept rates on the same per-token frame. None of the four
couples the analysis to cost-per-correct-answer or treats the drafter as
a \emph{workload-level} difficulty classifier.

\textbf{Cascade routing.} \citet{chen2023frugalgpt} builds a learned
cascade across API tiers and reports cost reductions of up to 98
percent at iso-accuracy when matching GPT-4 on HEAD-QA, SUBJ, and COQA. The paper reports the
empirical result; it does not derive the threshold under which the
cascade beats a single-tier baseline. \citet{zhan2026mpd2} extends the
cascade frame to human-in-the-loop deferral in medical imaging and
reports a Pareto frontier in F1, MCC, and cost on the REFUGE, CHAKSU,
and ORIGA glaucoma datasets. It inherits the same gap.
\citet{dooly2026inferencesim} provides background on profiling the
per-tier compute footprint that any cascade calibration must lean on.

\textbf{Adaptive sampling and exploration.} \citet{petullo2026veccisc}
prunes self-consistency candidates by semantic similarity and reports a
47 percent token reduction at iso-accuracy across math, chemistry,
biology, commonsense, and humanities benchmarks.
\citet{petullo2026casql} scales tree-search exploration breadth by
estimated task complexity and reports 51.72 percent on the challenging
tier of the BIRD development set using a GPT-4o-mini base. Both report empirical
token-cost reductions. Neither derives the condition under which
complexity-conditional compute is optimal. \citet{snell2024testtime}
supplies the per-difficulty-bin compute-vs-accuracy curves that the
calibration in \Cref{sec:experiments} leans on for the
workload-heterogeneity estimate.

\textbf{Early exit and adaptive depth.} \citet{schuster2022calm} derives
an exit threshold from a per-layer confidence signal inside the model
and reports up to 3x speedups on T5 backbones at iso-accuracy on the
CNN/DM, SQuAD, and WMT-EN-RO benchmarks. The exit threshold is a
\emph{local} version of difficulty-conditional compute: the classifier
is the per-layer confidence head, the routing decision is per-layer
rather than per-query, and the cost analysis is per-layer rather than
per-correct-answer.

\textbf{Cost-correct frame.} \citet{erol2026costofpass} introduces
\emph{Cost-of-Pass} as a per-accepted-correct-answer metric and reports
that the metric is dominated by token cost at frontier API tiers.
\citet{bhardwaj2026costcorrect} develops the multiplicative
decomposition $C = \mathrm{CPM} \cdot R \cdot (1 + \bar\rho) / \alpha$
that separates blended cost-per-million-tokens, the reasoning
multiplier $R$, the average rollout ratio $\bar\rho$, and the verifier
accept rate $\alpha$. \citet{bhardwaj2026alpha} shows that the partial
derivative of $C$ with respect to $\alpha$ dominates the partials with
respect to the other three components in production regimes.
Paper~\#1 of this wedge \citep{bhardwaj2026inferencefrontier} supplies
the channel-allocation threshold that the threshold here composes with.
The Cost-correct frame is what makes the threshold derivable: prior
frames (FLOPs-per-token, raw token cost) do not surface the
curvature-by-variance product that \eqref{eq:threshold} leans on.

The workload-heterogeneity numbers needed to estimate $\Delta$ in
production are reported in \citet{patel2024splitwise},
\citet{agrawal2024sarathi}, and \citet{slogard2026}; we use these in
\Cref{sec:discussion} to size the threshold against measured serving
workloads.

% =========================================================================
\section{Method. The routing-premium threshold}
\label{sec:method}

This section develops the threshold theorem. \Cref{sec:setup} sets up
the two-policy comparison. \Cref{sec:theorem} states and proves the
threshold to second order and reports the scope of validity.
\Cref{sec:elasticity} reports the threshold in elasticity form, which is
what the calibration in \Cref{sec:experiments} hooks into.
\Cref{sec:specializations} derives five corollaries, one per
allocation-rule class, each recovering a published instance.

\subsection{Setup}
\label{sec:setup}

The workload is a distribution $F(d)$ over latent difficulties $d \in
[0, 1]$. Cost-per-correct-answer at difficulty $d$ and compute level
$c$ is the Cost-correct expression of \citet{bhardwaj2026costcorrect},
\begin{equation}
C(c, d) \;=\; \frac{\mathrm{CPM} \cdot R(c, d) \cdot (1 + \rho(c, d))}
                    {\alpha(c, d)},
\label{eq:costcorrect}
\end{equation}
with $\mathrm{CPM}$ the blended cost per million tokens, $R$ the
reasoning multiplier, $\rho$ the rollout ratio, and $\alpha$ the
verifier accept rate, each potentially conditioning on $d$. Compute $c$
is the operationally controlled quantity: for speculative decoding it
is the drafter chunk size, for cascade routing it is the model-tier
index, for adaptive self-consistency it is the rollout count, for
complexity-aware exploration it is the tree-search breadth, and for
early-exit it is the exit-layer index. The provider chooses a policy
that maps difficulty information into $c$.

Two policies bracket the comparison.

\emph{Fixed-$c$.} The unconditional optimum
\begin{equation}
\bar c^* \;=\; \arg\min_c \, \mathbb{E}_F\bigl[ C(c, d) \bigr].
\label{eq:fixedc}
\end{equation}
The provider runs $\bar c^*$ on every query. This is the baseline. It
uses no difficulty information.

\emph{Router-$c(\cdot)$.} The provider obtains an estimator $\hat d$
from a difficulty classifier with calibration quality
\begin{equation}
\kappa \;=\; \frac{\mathrm{Var}\bigl(\, \mathbb{E}[c^*(d) \mid \hat d] \,\bigr)}
                  {\mathrm{Var}\bigl(\, c^*(d) \,\bigr)} \;\in\; [0, 1],
\label{eq:kappa}
\end{equation}
where $c^*(d) = \arg\min_c C(c, d)$ is the conditional optimum at
difficulty $d$. $\kappa$ is the explained-variance share of the
oracle-optimal compute that the classifier recovers from its estimate.
$\kappa = 1$ is the oracle, $\kappa = 0$ is uninformative. The provider
runs $c(\hat d) \in \arg\min_c \mathbb{E}[C \mid \hat d]$ and pays a
per-query classifier overhead
\begin{equation}
C_{\mathrm{cls}} \;=\; \gamma \cdot C(\bar c^*, \bar d),
\label{eq:gamma}
\end{equation}
with $\bar d$ the workload mean difficulty and $\gamma$ the
classifier-overhead ratio expressed as a dimensionless fraction of the
unconditional inference cost.

The four scalars $(\kappa, \Delta, \gamma, \bar c^*)$ summarize the
comparison. The remaining input is the curvature of $C$ in $c$ at the
unconditional optimum, which enters the threshold through the
dimensionless heterogeneity measure $\Delta$ defined below.

\subsection{Theorem. Routing premium}
\label{sec:theorem}

\begin{theorem}[Routing premium, local form]
\label{thm:routing}
Under \eqref{eq:costcorrect} through \eqref{eq:gamma}, at an interior
unconditional optimum $\bar c^*$ with $C \in C^3$ in $c$, the expected
per-query cost gap between the fixed-$c$ and router-$c(\cdot)$
policies admits the second-order expansion
\begin{multline}
\mathbb{E}_F\bigl[ C(\bar c^*, d) \bigr] \;-\;
  \mathbb{E}_F\bigl[ C(c(\hat d), d) \bigr] \\
\;=\; \tfrac{1}{2} \cdot \bigl|\, C''_{cc}(\bar c^*, \bar d) \,\bigr|
       \cdot \kappa \cdot \mathrm{Var}_d\bigl[ c^*(d) \bigr]
   \;-\; C_{\mathrm{cls}}
   \;+\; O\bigl(\, \| c(\hat d) - \bar c^* \|^3 \,\bigr).
\label{eq:expansion}
\end{multline}
Dividing through by the unconditional optimum cost $C(\bar c^*, \bar
d)$ and collecting terms gives the dimensionless form
\begin{equation}
\frac{\Pi}{C(\bar c^*, \bar d)} \;=\; \kappa \cdot \Delta \;-\; \gamma
  \;+\; O\bigl(\, \| c(\hat d) - \bar c^* \|^3 \,\bigr),
\label{eq:pi-norm}
\end{equation}
with $\Delta = |C''_{cc}(\bar c^*, \bar d)| \cdot \mathrm{Var}_d[c^*(d)]
/ (2 \cdot C(\bar c^*, \bar d))$ a dimensionless workload-heterogeneity
measure. Routing pays at the margin around $\bar c^*$ iff
\begin{equation}
\kappa \cdot \Delta \;>\; \gamma.
\label{eq:routing-premium}
\end{equation}
\end{theorem}

\begin{proof}[Proof sketch]
Expand $C(c(\hat d), d)$ around the unconditional optimum in $c$ and
take the workload expectation. The first-order term in $c(\hat d) -
\bar c^*$ vanishes because $\bar c^*$ is the unconditional minimum.
The second-order term picks up the curvature $C''_{cc}(\bar c^*, \bar
d)$ scaled by the squared deviation, which under the optimal-router
choice $c(\hat d) \in \arg\min_c \mathbb{E}[C \mid \hat d]$ has
expectation equal to the explained-variance share $\kappa$ of
$\mathrm{Var}_d[c^*(d)]$. Subtracting the classifier overhead
\eqref{eq:gamma} and dividing by the unconditional optimum cost yields
\eqref{eq:pi-norm}. The $O(\| \cdot \|^3)$ residual collects the
third-order curvature term, which is non-negligible at large $\| c(\hat
d) - \bar c^* \|$.
\end{proof}

\textbf{Scope of the theorem.} Condition \eqref{eq:routing-premium} is
\emph{local} around the unconditional optimum. It is necessary and
sufficient \emph{at the margin}. For large deviations the third-order
curvature term in \eqref{eq:pi-norm} can dominate and reverse the sign
of the gap, so the threshold does not extend to a global guarantee for
aggressive routing policies. The five published patterns we calibrate
in \Cref{sec:experiments} operate inside this local regime; the cascade
specialization (\Cref{sec:specializations}) is the closest to the
boundary because a binary tier choice moves $c$ a long way from $\bar
c^*$. We carry the third-order correction explicitly through the
cascade rows in \Cref{sec:experiments} and the serving-stack discussion
in \Cref{sec:serving}.

The economic content of \eqref{eq:routing-premium} is a
curvature-by-variance product against a fixed overhead. $\kappa$ is a
calibration quantity; $\Delta$ is a workload quantity; $\gamma$ is a
stack quantity. Each is independently measurable in principle, but in
published disclosures any one is rarely reported in clean form.
\Cref{sec:elasticity} reformulates \eqref{eq:routing-premium} so the
calibration in \Cref{sec:experiments} can hook into the disclosed
numbers each system \emph{does} report.

\subsection{The threshold in elasticity form}
\label{sec:elasticity}

Let $\Pi = \kappa \cdot \Delta - \gamma$ denote the normalized routing
premium from \eqref{eq:pi-norm}. The elasticities of $\Pi$ with respect
to the three observables are
\begin{equation}
\frac{\partial \log \Pi}{\partial \log \kappa} \;=\;
  \frac{\kappa \Delta}{\kappa \Delta - \gamma},
\qquad
\frac{\partial \log \Pi}{\partial \log \Delta} \;=\;
  \frac{\kappa \Delta}{\kappa \Delta - \gamma},
\qquad
\frac{\partial \log \Pi}{\partial \log \gamma} \;=\;
  \frac{-\gamma}{\kappa \Delta - \gamma}.
\label{eq:elasticity}
\end{equation}

The two elasticities in $\kappa$ and $\Delta$ are equal and positive,
with magnitude diverging at the threshold $\kappa \Delta = \gamma$. The
elasticity in $\gamma$ is negative with absolute magnitude $\gamma /
(\kappa \Delta)$ times the other two. Three readings of
\eqref{eq:elasticity} hook into the calibration in
\Cref{sec:experiments}.

First, the elasticity form lets the calibration report a
\emph{disclosed change} in one observable rather than a point estimate
of all three. Each published system in \Cref{sec:experiments} discloses
at least one of $\kappa$ (drafter accept rate, routing accuracy,
breadth-vs-bin schedule), $\Delta$ (per-tier prices, per-bin compute,
accept-rate-curve curvature), or $\gamma$ (drafter cost share,
router-call latency, layer-confidence-head FLOPs). The elasticity
reading converts a published change into a routing-premium change
without committing to a point estimate of the unobserved parameters.

Second, the elasticity is \emph{divergent} at the threshold. Operating
points close to $\kappa \Delta = \gamma$ are sensitive: small
disclosure errors flip the sign of $\Pi$. \Cref{sec:experiments}
reports the elasticity bar at each calibration row and flags CALM as
the natural sensitivity case.

Third, the equal-magnitude positive elasticities in $\kappa$ and
$\Delta$ mean that calibration improvements and workload-heterogeneity
increases buy the same routing premium per log-point. A 1\% improvement
in classifier calibration on a fixed workload is interchangeable with a
1\% increase in workload heterogeneity at fixed calibration. This is
the operational reading: serving stacks can lift $\Pi$ either by
sharpening the difficulty classifier or by serving more heterogeneous
workload mixes.

A brief reading of the three parameters in turn.

\emph{$\kappa$, classifier calibration.} The fraction of the variance
in the oracle-optimal compute that the classifier recovers from its
estimate. $\kappa = 1$ for an oracle. Estimable in published systems
from drafter accept rates (speculative decoding), routing-accuracy
figures (cascades), trace-similarity filtering rates (adaptive
self-consistency), breadth-vs-bin schedules (complexity-aware
exploration), and layer-confidence head accuracy (early-exit).

\emph{$\Delta$, compute-variance heterogeneity.} Large when the
workload mixes easy and hard queries, the accept-rate curve $\alpha(c,
d)$ is concave in $c$, and the conditional optimum $c^*(d)$ moves
substantially across difficulty bins. Small for homogeneous workloads.
The dimensionless form $\Delta = |C''_{cc}| \cdot \mathrm{Var}_d[c^*(d)]
/ (2C)$ has natural decomposition into a curvature factor and a
variance factor; the curvature factor is set by the local second
derivative of cost in compute, the variance factor by the operational
workload mix.

\emph{$\gamma$, classifier overhead.} Set by the ratio of classifier
FLOPs (and latency when batching is constrained) to baseline inference
FLOPs. Typical values cluster in $10^{-3}$ to $10^{-1}$ for
transformer-based drafters and routers in 2026 disclosures. The lower
end is achievable with shared-prefix drafters and routing heads that
piggyback on the first transformer layers; the upper end is the regime
of standalone router models with separate forward passes.

\subsection{Specializations}
\label{sec:specializations}

Five corollaries of \Cref{thm:routing} recover the published instances,
one per allocation-rule class.

\begin{corollary}[Speculative decoding]
\label{cor:spec}
In the per-token frame with drafter chunk size $k$ and drafter accept
rate $a$, condition \eqref{eq:routing-premium} reduces to the
classical speculative-decoding speedup condition. The classifier is
the drafter, $\kappa$ is monotone-increasing in $a$, $\gamma$ is the
drafter-to-verifier FLOPs ratio, and $\Delta$ is the per-token
compute-variance from the verifier accept-rate curve. The Leviathan
speedup expression falls out as the special case of an i.i.d.
token-difficulty distribution with a uniform drafter calibration.
\end{corollary}

The within-sequence application is the operative point: $\Delta$ is
\emph{per-token} heterogeneity in compute, not per-query heterogeneity.
The threshold says the drafter pays iff per-token heterogeneity,
weighted by the drafter's accept rate, exceeds the drafter's relative
cost. This is the form practitioners already use for speculative
decoding \citep{leviathan2023speculative, cai2024medusa};
\Cref{thm:routing} nests it.

\begin{corollary}[Cascade routing]
\label{cor:cascade}
In the two-tier system with small-tier compute $c_s$ and large-tier
compute $c_\ell$, the optimal router policy is binary, $c(\hat d) \in
\{c_s, c_\ell\}$. Condition \eqref{eq:routing-premium} reduces to a
per-bin break-even on tier prices: a query is sent to the large tier
iff the per-difficulty-bin gain in $\alpha$ exceeds the price gap. The
threshold is the workload-averaged version of the per-bin break-even,
with the third-order correction non-negligible when the per-tier price
gap is large.
\end{corollary}

Cascade routing is where the local-margin assumption from
\Cref{sec:theorem} binds hardest. The binary policy moves $c$ a long
way from $\bar c^*$ on every query, not just at the boundary. The
third-order term in \eqref{eq:pi-norm} is therefore non-trivial for the
cascade specialization and we carry it explicitly in the FrugalGPT and
MPD$^2$-Router calibration rows in \Cref{sec:experiments}.

\begin{corollary}[Adaptive self-consistency]
\label{cor:adaptive}
With compute parameterized by the rollout count $\rho$ and the
classifier the trace-similarity filter that prunes degenerate traces,
condition \eqref{eq:routing-premium} reduces to a per-query break-even
on the rollout count. The classifier $\kappa$ is set by the fraction of
degenerate traces correctly identified; $\Delta$ is the per-query
compute-variance of the cost-correct optimum rollout count; $\gamma$
is the embedding-and-similarity cost per trace.
\end{corollary}

The VecCISC 47 percent token reduction at iso-accuracy
\citep{petullo2026veccisc} implies $\kappa \cdot \Delta$ near 0.5
across the five reported domains (math, chemistry, biology,
commonsense, humanities) once $\gamma$ is read off the disclosed
embedding-network cost. \Cref{sec:experiments} reports the calibration.

\begin{corollary}[Complexity-aware exploration]
\label{cor:complexity}
With compute parameterized by exploration breadth $k$ in tree-search
or sampling and the classifier a difficulty estimator that scales
breadth per query, the optimal router rule is $k(\hat d) = k_0 \cdot
\exp(\beta \hat d)$ with $\beta$ pinned by $\kappa \cdot \Delta$ at
the workload mean.
\end{corollary}

CA-SQL's breadth schedule on the challenging tier of BIRD
\citep{petullo2026casql} recovers as the special case of this rule
with $\beta$ inferred from the disclosed breadth-vs-bin schedule. The classifier here is the
difficulty-bin assignment from the schema-and-question encoder;
$\gamma$ is set by the per-query encoder pass.

\begin{corollary}[Early exit]
\label{cor:early}
In the per-layer frame with compute parameterized by exit-layer index
$\ell$ and the classifier the per-layer confidence head, condition
\eqref{eq:routing-premium} reduces to a per-layer per-token exit
condition. The classifier $\kappa$ is bounded by the per-layer
confidence-head calibration on the training distribution; $\Delta$ is
bounded by the depth-dependent curvature of the accept-rate curve and
is small in absolute terms; $\gamma$ is set by the per-layer
confidence-head FLOPs.
\end{corollary}

Early exit recovers the CALM exit rule of \citet{schuster2022calm} as
the special case of \Cref{cor:early} with the confidence-head signal as
the classifier and a single calibration constant fit per workload.
Because $\Delta$ is bounded by depth-dependent curvature, the operating
point sits closest to the threshold $\kappa \Delta = \gamma$ among the
five specializations. CALM is the natural sensitivity case in
\Cref{sec:experiments}.

The five corollaries are not independent: each is a coordinate chart on
the same threshold \eqref{eq:routing-premium}, with the role of $c$,
the classifier, and the workload distribution specialized to the
allocation-rule class. The unifying claim of the paper is that the five
literatures are studying one inequality.

% =========================================================================
\section{Experiments. Calibration from six published systems}
\label{sec:experiments}

We calibrate \Cref{thm:routing} against six published operating points.
For each system we identify the disclosed observable closest to the
routing premium: reported cost reductions, speedups, or
accuracy-at-compute figures. We map that observable to the
routing-premium product $\kappa\Delta$ using the corollary from
\Cref{sec:specializations}, and bound $\gamma$ from the disclosed
classifier overhead. The routing premium $\Pi = \kappa\Delta - \gamma$
is then a disclosed-derived band rather than a point estimate;
elasticity error bars from \eqref{eq:elasticity} report the local
sensitivity.

The six systems are grouped by allocation-rule class.
\Cref{tab:calibration} at the end of the section summarizes the
calibration. All six rows have $\Pi > 0$; the bands vary considerably
in width.

\textbf{Speculative decoding: Leviathan and Medusa.}
\citet{leviathan2023speculative} report 2 to 3 times end-to-end
inference speedups on T5-class models with drafter accept rates in the
$0.6$ to $0.8$ range and a drafter-to-verifier FLOPs ratio of roughly 1
to 5 percent. \citet{cai2024medusa} report a 2.2 times speedup for
Medusa-1 and 2.3 to 3.6 times for Medusa-2 on LLaMA-class backbones;
FLOPs-ratio disclosure is not in the paper abstract and we bound it
from the Medusa-1 architecture description in the body (a small number
of multi-head drafters added on top of the base model). In
\Cref{cor:spec}, $\gamma$ is the drafter FLOPs ratio and $\kappa\Delta
- \gamma = 1 - 1/S$ where $S$ is the per-token speedup. For $S = 2$ to
$3$, this gives $\Pi = 0.50$ to $0.67$. Bounding $\gamma \in [0.01,
0.05]$ implies $\kappa\Delta \in [0.51, 0.72]$. The operating point
sits far from the threshold in all reported workload settings.
Elasticity magnitude is moderate: a 10 percent degradation in drafter
accept rate shifts $\Pi$ by approximately 0.07 to 0.12 in this band.

\textbf{Cascade routing: FrugalGPT.} \citet{chen2023frugalgpt} reports
cost reductions of up to 98 percent at iso-accuracy when matching GPT-4
on HEAD-QA, SUBJ, and COQA via a learned cascade across API tiers. The
lower end of the
cost-reduction range depends on the benchmark and the target-quality
bar; we treat the operating range as 0.40 to 0.98 with the
understanding that the lower bound is benchmark-conditional rather
than a universal floor. The FrugalGPT router is a trained prompt
scorer with overhead estimated at less than 1 percent of the cost of a
large-tier call (a small classification head over the prompt
embedding). Taking $\gamma \in [0.001, 0.010]$ and reading $\Pi$ from
the reported cost-reduction fraction gives $\kappa\Delta \in [0.40,
0.99]$ across the three datasets. The wide band reflects the range across datasets;
the binary tier policy warrants the third-order correction flagged in
\Cref{cor:cascade}. Even at the conservative end ($\Pi \approx 0.40$,
HEAD-QA), the operating point sits well into the positive side.
Elasticity magnitude is low: a 10 percent change in router accuracy
shifts $\Pi$ by approximately 0.04 to 0.10.

\textbf{Cascade routing: MPD$^2$-Router.} \citet{zhan2026mpd2} reports
that the framework is Pareto-optimal in F1, MCC, and cost on all three
cross-national glaucoma cohorts (REFUGE, CHAKSU, ORIGA) at a
moderate deferral rate. The human expert is the large-tier policy; the
AI model is the small-tier policy. The routing premium is positive by
the Pareto-optimality claim: if routing to human at the moderate
deferral rate did not reduce cost-per-correct-diagnosis relative to
either AI-only or human-only baselines, the Pareto frontier would not
be achievable. Exact values of $\kappa$, $\Delta$, and $\gamma$ are not
fully disclosed in the abstract; we treat this row as a qualitative
sign-confirmation rather than a precise calibration. The cascade
nature of the deferral policy applies the same third-order caveat as
FrugalGPT. We assign the widest elasticity uncertainty band among the
six rows on account of the thin $\gamma$ disclosure and the binary
deferral structure.

\textbf{Adaptive self-consistency: VecCISC.}
\citet{petullo2026veccisc} reports a 47 percent total token reduction
at iso-accuracy across five benchmark domains. The classifier is a
semantic-similarity filter over reasoning traces; the
degenerate-trace filter is a lightweight sentence-embedding
comparison (embedding models in the $10^8$ parameter range,
approximately $0.02$ to $0.05$ of a GPT-4o-mini forward pass in FLOPs).
Taking $\gamma \in [0.02, 0.05]$ and reading $\Pi \approx 0.47$ from
the reported token reduction gives $\kappa\Delta \in [0.49, 0.52]$.
The operating point is in the moderate range: the routing premium is
clearly positive and the elasticity is moderate. A 10 percent drop in
trace-filtering accuracy shifts $\Pi$ by approximately 0.05 to 0.08.

\textbf{Complexity-aware exploration: CA-SQL.}
\citet{petullo2026casql} reports 51.72 percent execution accuracy on
the challenging tier of the BIRD development set using GPT-4o-mini
with a difficulty-adaptive breadth schedule, outperforming approaches that
use GPT-4 at fixed breadth (which implies the
adaptive-small-model policy achieves lower cost-per-correct-answer
than a non-adaptive larger-model policy). The difficulty estimator is
a schema-and-question encoder running ahead of the tree search; at
roughly 5 to 10 percent of a GPT-4o-mini call, $\gamma \in [0.05,
0.10]$. The exact value of $\kappa\Delta$ requires cost disclosure
that the paper does not provide; however the accuracy dominance over
fixed-breadth larger models implies $\Pi > 0$ under the Cost-correct
interpretation (same or lower cost, higher $\alpha$). We report this
row as sign-confirmed with thin cost disclosure and assign a wide
$\Delta$ uncertainty band. The elasticity reading is therefore wide,
and we do not report a point estimate of $\kappa\Delta - \gamma$ for
this row.

\textbf{Early exit: CALM.} \citet{schuster2022calm} reports up to 3x
inference speedups on T5 backbones for CNN/DM summarization, SQuAD
question answering, and WMT-EN-RO translation. The per-layer
confidence head is a single linear layer over the hidden state at each
Transformer depth; its FLOPs overhead $\gamma_\ell \in [0.01, 0.05]$
per layer. Per-layer $\Delta$ is bounded by the depth-curvature of the
per-token accept-rate curve, which is small compared to the per-query
heterogeneity in the cascade and self-consistency rows. The observed
speedup of 1.5 to 2 times on the average task (the 3 times figure is
the SQuAD peak) implies per-layer $\Pi \approx 0.10$ to $0.20$, i.e.,
$\kappa\Delta \in [0.11, 0.25]$ when $\gamma \in [0.01, 0.05]$. CALM
sits closest to the threshold $\kappa\Delta = \gamma$ of the six rows.
The elasticity is divergent near the threshold: a 20 percent drop in
per-layer confidence calibration ($\kappa$) shifts $\Pi$ by
approximately $0.8 \times \Pi$, which could flip the sign on
low-information layers. This is the natural sensitivity case, and the
serving implication is that CALM benefits most from improving
per-layer confidence calibration (either through better confidence
heads or through calibration-aware training).

\begin{figure}[t]
  \centering
  \includegraphics[width=0.80\linewidth]{figures/fig1_isoclines.png}
  \caption{Routing-premium threshold in classifier-calibration and
  workload-heterogeneity space. Each curve is the zero-premium isocline
  $\kappa\Delta = \gamma$ for a fixed classifier overhead $\gamma$.
  Above each curve the routing premium $\Pi = \kappa\Delta - \gamma$ is
  positive and routing reduces cost-per-correct-answer relative to the
  unconditional optimum. The three $\gamma$ values span the range from
  lightweight routing heads ($\gamma = 0.02$) to moderately expensive
  standalone classifiers ($\gamma = 0.10$). Reproducibility:
  \texttt{paper-2/figures/fig1\_isoclines.py}.}
  \label{fig:isoclines}
\end{figure}

\begin{figure}[t]
  \centering
  \includegraphics[width=0.85\linewidth]{figures/fig2_calibration.png}
  \caption{Six published systems plotted in $(\kappa, \Delta)$ space.
  Error bars are elasticity bands from \eqref{eq:elasticity}:
  horizontal bars span the range of $\kappa$ implied by a $\pm 10$
  percent change in the primary disclosed observable (drafter accept
  rate, cost-reduction fraction, token-reduction fraction, or speedup
  ratio); vertical bars span the corresponding range of $\Delta$
  holding $\kappa\Delta$ fixed at the disclosed midpoint. Each dashed
  curve is the zero-premium isocline $\kappa\Delta = \gamma$ for the
  system's $\gamma$ midpoint. All six systems sit above their own
  threshold line. CALM is closest to its threshold (smallest $\Pi =
  0.15$) and is labeled as the sensitivity case. Reproducibility:
  \texttt{paper-2/figures/fig2\_calibration.py}.}
  \label{fig:calibration}
\end{figure}

\textbf{Calibration table.} \Cref{tab:calibration} collects the six
rows. Columns report the disclosed observable, the implied routing
premium $\Pi = \kappa\Delta - \gamma$, a bounded estimate of $\gamma$,
the implied product $\kappa\Delta$, and the elasticity sensitivity
label (Low, Moderate, or High).

\begin{table}[h]
\centering
\caption{Routing-premium calibration across six published systems.
$\Pi = \kappa\Delta - \gamma$ derived from disclosed cost-reduction,
speedup, or accuracy figures. $\gamma$ bounded from disclosed
classifier overhead. $\kappa\Delta$ is the derived product; individual
$\kappa$ and $\Delta$ decomposition requires workload characterization
not disclosed in any of the six papers. Elasticity: sensitivity of
$\Pi$ to a 10 percent change in the disclosed primary observable, per
\eqref{eq:elasticity}. Wide bands in the MPD$^2$-Router and CA-SQL rows
reflect thin cost disclosure.}
\label{tab:calibration}
\footnotesize
\setlength{\tabcolsep}{4pt}
\begin{tabular}{l l l l l l}
\toprule
System & Class & Disclosed metric & $\hat\gamma$ & $\kappa\Delta$ & $\Pi$ band \\
\midrule
Leviathan / Medusa & Spec. decoding & 2--3$\times$ speedup &
  0.01--0.05 & 0.51--0.72 & 0.50--0.67 \\
FrugalGPT & Cascade & 40--98\% cost red. &
  0.001--0.010 & 0.40--0.99 & 0.40--0.98 \\
MPD$^2$-Router & Cascade & Pareto F1-MCC-cost &
  0.01--0.03 & $> \hat\gamma$ & $>0$ (thin) \\
VecCISC & Self-consistency & 47\% token red. &
  0.02--0.05 & 0.49--0.52 & $\approx 0.47$ \\
CA-SQL & Complexity-aware & 51.72\% BIRD (challenging) &
  0.05--0.10 & $> \hat\gamma$ & $>0$ (thin) \\
CALM & Early exit & 1.5--3$\times$ speedup &
  0.01--0.05 & 0.11--0.25 & 0.10--0.20 \\
\bottomrule
\end{tabular}
\end{table}

All six operating points sit on the positive side of the threshold. The
distribution is right-skewed: FrugalGPT occupies the widest band (0.40
to 0.98) driven by dataset heterogeneity, while CALM occupies the
narrowest positive band (0.10 to 0.20) driven by the per-layer
constraint on $\Delta$. The CALM band's lower end at $\Pi \approx
0.10$ is the closest to the threshold among the six, which is why
\Cref{sec:serving} uses CALM as the leading example of the sensitivity
tradeoff. \Cref{fig:calibration} plots the six systems in $(\kappa,
\Delta)$ space with elasticity bars and the system-specific $\gamma$
threshold lines.

% =========================================================================
\section{Discussion}
\label{sec:discussion}

\subsection{Serving-stack design under measured workload heterogeneity}
\label{sec:serving}

The threshold \eqref{eq:routing-premium} is a design criterion, not
just a condition. When $\Pi > 0$ is comfortable (Leviathan, FrugalGPT,
VecCISC), the difficulty classifier earns its cost by wide margins
across workload compositions; adding or removing it from the serving
path has modest impact on cost-per-correct-answer. When $\Pi$ is small
(CALM, thin-disclosure cascades), the serving designer should treat
the classifier as a continuously monitored component: a classifier
that was calibrated on a historical workload mix can fall below the
threshold if the live workload drifts toward homogeneity.

This has a concrete implication for infrastructure.
\citet{patel2024splitwise} measure prefill-decode heterogeneity in a
production LLM cluster and report that the token-distribution variance
across queries spans more than two orders of magnitude.
\citet{agrawal2024sarathi} measure latency sensitivity to batching
policy and show that the variance in query length (a proxy for
difficulty) is large enough to justify dynamic chunked-prefill
scheduling. \citet{slogard2026} studies autotuning of serving configurations
under SLO constraints, providing the empirical setting where
$\gamma$ and per-tier compute costs are measured. Taken together, these three sources imply that
$\Delta$ in large-scale LLM serving is well above the threshold for
the current generation of lightweight routers ($\gamma \approx 0.01$
to $0.03$), supporting the classification of the serving problem as
firmly $\Pi > 0$.

The operational recommendation is: when workload heterogeneity
(measured as the variance of the optimal-compute-per-query
distribution across a representative traffic sample) exceeds $\gamma /
\kappa_0$, where $\kappa_0$ is the estimated calibration of the
available difficulty classifier, providers should expose the
classifier as a first-class API parameter rather than keeping it
internal. Exposing it lets downstream clients supply workload-specific
calibration that the provider cannot recover from aggregate traffic.

\subsection{Why frontier reasoning APIs are converging on tier menus}
\label{sec:tiermenu}

OpenAI's o-series, Anthropic's Claude Sonnet and Opus tiers, and
Google's Gemini Pro and Flash all expose an explicit per-query budget
knob or model-tier choice. None of the three providers published a
derivation of this choice. The threshold \eqref{eq:routing-premium}
provides a post-hoc explanation: if $\gamma$ is low enough (i.e., a
routing head or a per-query budget parameter add negligible marginal
cost), and if the workload heterogeneity $\Delta$ at frontier scale is
large enough (which the production serving studies above support),
then the routing premium $\Pi > 0$ across the space of realistic
provider workloads. The tier-menu architecture is the market-level
response to \eqref{eq:routing-premium}: rather than routing internally
at the provider level, providers expose the routing decision to
clients who hold private workload information, trading the loss of
provider-side $\kappa$ optimization for the gain of client-side
workload disclosure.

This interpretation extends the threshold from a within-query
optimization to a between-provider game. The tier menu reduces the
effective $\gamma$ to zero (the client chooses the tier at query time
with no additional overhead), and the client's task-difficulty
information replaces the trained classifier as the source of $\kappa$.
When $\kappa$ from client selection exceeds the provider's classifier
$\kappa$, the client-routing regime dominates the provider-routing
regime. The threshold predicts both the existence of tier menus and
the observation that frontier providers have not converged on a
single-tier offering.

\subsection{Composition with Paper~\#1}
\label{sec:composition}

Paper~\#1 of this wedge \citep{bhardwaj2026inferencefrontier} derives
the threshold for \emph{which channel} (training versus inference) the
next compute dollar should go. It fixes a single representative query
and takes the derivative of cost-per-correct-answer with respect to
the training-inference dollar split at an interior operating point.
The result is a switching condition $(\eta_\alpha^\rho -
1)/\eta_\alpha^T > 1/\mu$, where $\mu$ is the inference-to-training
cost ratio and $\eta_\alpha^\rho$, $\eta_\alpha^T$ are the
accept-rate elasticities with respect to rollout count and training
compute.

Paper~\#2 (this work) derives the threshold for \emph{how to allocate}
a fixed inference budget across a heterogeneous workload. The two
thresholds are orthogonal: Paper~\#1 asks whether to put the next
dollar in inference at all; Paper~\#2 asks, given that some dollars
are in inference, whether a calibrated difficulty classifier improves
the allocation. They compose multiplicatively. The combined production
cost satisfies
\begin{equation}
C_{\mathrm{total}} \;=\; C(\bar c^*, \bar d)
  \cdot \bigl[ 1 - \Pi_1 \bigr]^{\mathbf{1}[\text{ch = inference}]}
  \cdot \bigl[ 1 - \Pi_2 \bigr]^{\mathbf{1}[\kappa\Delta > \gamma]},
\label{eq:composition}
\end{equation}
where $\Pi_1$ is the Paper~\#1 routing premium (inference channel
relative to training channel) and $\Pi_2 = \kappa\Delta - \gamma$ is
the Paper~\#2 routing premium. The two indicators are independent: the
inference-channel decision (Paper~\#1) gates on query difficulty and
training-cost structure; the workload-routing decision (Paper~\#2)
gates on workload heterogeneity and classifier overhead.
\Cref{fig:composition} plots both thresholds on the same axes, with
the four quadrants labeled by the implied allocation regime.

\begin{figure}[t]
  \centering
  \includegraphics[width=0.85\linewidth]{figures/fig3_composition.png}
  \caption{Composition of the Paper~\#1 channel-allocation threshold
  (vertical line at $\mu = \mu^*$: inference-to-training cost ratio vs.
  the Paper~\#1 switching threshold) and the Paper~\#2 workload-routing
  threshold (horizontal line at $\kappa\Delta = \bar\gamma$). The four
  quadrants correspond to the four combinations of training-dominant
  or inference-dominant channel choice and flat or routed
  within-inference allocation. Frontier reasoning tiers sit in the
  upper-right quadrant: inference-dominant and heterogeneous enough to
  route. Commodity tiers sit in the lower-right: inference-dominant
  but flat in compute allocation. The six Paper~\#2 calibration points
  are projected onto the $\kappa\Delta$ axis. Reproducibility:
  \texttt{paper-2/figures/fig3\_composition.py}.}
  \label{fig:composition}
\end{figure}

The composition has a serving implication. A provider that has crossed
Paper~\#1's threshold (i.e., it is already cost-optimal to invest in
inference-time scaling) will also want to cross Paper~\#2's threshold
if the workload is heterogeneous enough. The two conditions can both
be satisfied at the same operating point, and in the production
workloads we examine ($\Delta \gg \gamma / \kappa$, $\eta_\alpha^\rho -
1 \gg 1/\mu$) they are both satisfied simultaneously. The combined
cost reduction is multiplicative and larger than either reduction
alone.

% =========================================================================
\section{Conclusion}
\label{sec:conclusion}

The routing premium $\kappa\Delta > \gamma$ is positive at the margin
around the unconditional optimum when the classifier calibration and
the workload heterogeneity together exceed the classifier overhead. We
derive the condition from the Cost-correct framework, show it nests
the five major published instances of difficulty-conditional compute
as corollaries, and calibrate it against six operating points spanning
all five classes. Every calibrated operating point sits on the
positive side. CALM, as the early-exit representative, sits closest to
the threshold: its per-layer $\Delta$ is bounded by depth-curvature,
making it the sensitivity case that constrains the useful operating
range of exit confidence calibration.

The derivation has two open edges. First, $\kappa$ at production scale
is not directly observable from public APIs: the explained-variance
calibration of a provider's internal difficulty classifier is not
disclosed in any of the six papers we calibrate, and we infer it from
proxy observables. Second, the second-order local result is sufficient
when the classifier policy stays close to the unconditional optimum,
but cascade and deferral systems that make large discrete jumps in
compute can violate the local approximation; a global routing-premium
result (incorporating all orders of the Taylor expansion) remains
open.

We invite serving providers to disclose routing-accuracy distributions
alongside cost-reduction reports. A disclosed $\kappa$ on a
representative workload sample would let independent researchers
verify or falsify the threshold directly, rather than relying on the
elasticity reading from proxy observables. That disclosure would also
distinguish the source of cost reductions in deployed tier-menu
systems: whether the gains come from calibration ($\kappa$ close to
1), from workload heterogeneity ($\Delta$ large), or from a fortuitous
combination of both.

\section{Cite this article}
\label{sec:bibtex}

\begin{verbatim}
@article{bhardwaj2026routingpremium,
  author = {Manu Bhardwaj},
  title  = {The Routing Premium.
            An Economic Threshold for
            Difficulty-Conditional Inference Compute},
  journal = {arXiv preprint},
  year   = {2026},
  url    = {https://ifitsmanu.com/papers/
             the-routing-premium},
}
\end{verbatim}

% =========================================================================
\bibliography{references}

\papercolophon

\end{document}
