\documentclass[11pt]{article}

\usepackage{../../template/ifitsmanu-paper}

%% --- Paper metadata ---
\title{Disaggregated or Colocated?\\
       The Cost-Frontier of LLM Serving Under SLO Contracts.}
\author{Manu Bhardwaj}
\date{May 2026}

\renewcommand{\shortauthor}{Bhardwaj}
\renewcommand{\shorttitle}{The Cost-Frontier of LLM Serving}

%% Notational shortcuts for the cost decomposition.
\newcommand{\CPM}{\mathrm{CPM}}
\newcommand{\CPMserved}{\mathrm{CPM}_{\mathrm{served}}}
\newcommand{\TTFT}{\mathrm{TTFT}}
\newcommand{\TPOT}{\mathrm{TPOT}}
\newcommand{\KVreq}{\mathrm{KV}_{\mathrm{req}}}

\begin{document}

\maketitle

\begin{abstract}
LLM serving in 2026 is not a single architecture. Colocated continuous
batching, chunked-prefill colocation, and prefill/decode disaggregation
each report goodput wins on different workload mixes against different
baselines. Production teams pick architectures without a frontier to
point at. We develop a closed-form decomposition of cost per
SLO-compliant served token into a prefill term, a decode term, and a
KV-transfer tax that applies only in disaggregated mode. We re-derive
published throughput numbers from five 2023--2025 systems papers into a
common frame. We plot the first cross-system Pareto frontier under
explicit p99 TTFT and p99 TPOT contracts. We solve for the break-even
surface between colocated and disaggregated architectures as a function
of input/output ratio, arrival rate, KV-transfer bandwidth, and SLO
slack. The frontier partitions. Disaggregation dominates the
prefill-heavy long-context region. Chunked-prefill colocation dominates
the decode-heavy short-context region. The crossover is sensitive to
KV-transfer bandwidth and shifts visibly between A100, H100, and H200
deployments.
\end{abstract}

% =========================================================================
\section{Why a frontier and not a winner}
\label{sec:why}

Iteration-level scheduling, introduced by Orca \citep{yu2022orca}, is
the primitive the modern serving stack is built on. Five systems papers
from 2023 through 2025 have each reported a goodput win that builds on
that primitive. PagedAttention and vLLM \citep{kwon2023pagedattention}
paged the KV cache and lifted realized batch size. Sarathi-Serve
\citep{agrawal2024sarathi} chunked prefill into smaller units
interleaved with decode iterations. DistServe \citep{zhong2024distserve}
split prefill and decode across separate worker pools. Splitwise
\citep{patel2024splitwise} did the same on Azure-scale hardware.
Mooncake \citep{qin2025mooncake} disaggregated at production scale at
Moonshot. Each report is internally honest. Each is a regional claim,
not a global one.

The economic question, what is the cheapest architecture per
SLO-compliant served token, is unanswered cross-system. Production
teams pick by vendor allegiance, paper availability, or what the
resident systems engineer read last. The serving choice is load-bearing
on annual GPU spend. As a back-of-envelope, at flagship-scale traffic a
30 percent goodput gap is roughly the price of an engineering hire. The
choice deserves a frontier.

The systems community has the throughput numbers. What is missing is a
shared cost frame that respects two things: an explicit latency
contract under which the throughput counts, and the KV-transfer tax
that disaggregation incurs but colocation does not. We supply one. We
use it to plot the first cross-system Pareto frontier of LLM serving
architectures and to solve for the break-even surface between colocated
and disaggregated regimes. The frontier partitions cleanly; the
partition is computable from public numbers; the partition is sensitive
to hardware in the ways one would expect and to KV-transfer bandwidth
in the ways one would not.

The contribution is three results. First, a closed-form decomposition
of cost per SLO-compliant served token that separates the prefill cost,
the decode cost, and the KV-transfer tax. Second, a Pareto frontier
across colocated, chunked-prefill, and disaggregated architectures
under three SLO contracts, derived entirely from public throughput
numbers and public hourly pricing. Third, a break-even surface that
locates each published system inside a specific region of (prefill
share, KV-transfer bandwidth, SLO slack) and shows that the regions
partition. No paper dominates the frontier. The frontier dominates the
papers.

The paper carries the Field Note \#1 frame
\citep{bhardwaj2026inferencestack} from the model-and-pricing layer to
the serving-architecture layer. Verified Capability per Dollar (VCpD)
was the right unit at the API level. Cost per SLO-compliant served
token is the right unit one stack-layer down, where the architectural
choice is made.

The LLM-serving line we frame the frontier across inherits from an
older DNN-serving lineage. Clipper \citep{crankshaw2017clipper}
introduced the model-server abstraction and the SLO-aware admission
control that the modern LLM systems carry forward. INFaaS
\citep{romero2021infaas} generalized model selection at serving time.
Shepherd \citep{zhang2023shepherd} formalized variance-aware DNN
serving under shared infrastructure. AlpaServe \citep{li2023alpaserve}
developed the statistical-multiplexing approach to model parallelism
that the LLM disaggregation papers re-frame as phase splitting. FlexGen
\citep{sheng2023flexgen} is the parallel single-GPU offloading line for
LLM serving that complements the multi-GPU frontier this paper
develops. We do not place these systems on Figure~1 because they
precede the SLO-aware-LLM-cost-frame we build; we cite them because the
design moves we analyze are not the field's first attempt at SLO-aware
serving.

% =========================================================================
\section{The unit: cost per SLO-compliant served token}
\label{sec:unit}

Define a serving system $S$ at fixed hardware $H$ and model $M$. Let
$\lambda$ be the request arrival rate and $D$ be the workload
distribution over (input length, output length) pairs. Let the SLO
contract be the pair $(\TTFT_{99}, \TPOT_{99})$: a request is
SLO-compliant if its time-to-first-token is at most $\TTFT_{99}$ and
its mean time-per-output-token is at most $\TPOT_{99}$. The realized
SLO-compliant throughput $G(S, H, M, \lambda, D, \mathrm{SLO})$ is the
rate at which the system emits tokens that belong to SLO-compliant
requests; it is bounded above by raw throughput and equal to it only
when no request misses the contract.

Cost per million SLO-compliant served tokens is
\begin{equation}
\CPMserved(S, H, M, \lambda, D, \mathrm{SLO})
  \;=\; \frac{c_H \cdot N_H}{G(S, H, M, \lambda, D, \mathrm{SLO}) / 10^6},
\label{eq:cpm-served}
\end{equation}
where $c_H$ is the hourly price of one accelerator of type $H$ and
$N_H$ is the number of accelerators the system uses to sustain $G$. The
numerator is the per-hour deployment bill. The denominator is what the
deployment delivers under the contract. The ratio is the cost of one
million SLO-compliant tokens.

Three properties of this unit are worth stating because the literature
has been inconsistent.

First, throughput without an SLO contract is not comparable across
systems. Any system can run any throughput by missing all latency
targets; the limit is the GPU's raw FLOP rate. Throughput-with-SLO is
the operative quantity. Goodput as defined in DistServe
\citep{zhong2024distserve} is the closest published unit; we extend it
by carrying the contract forward as part of the unit's identity, not as
a footnote.

Second, a token that violates the SLO is not delivered. It is paid for
twice. Once when the system burns compute producing it, and again when
the user-facing layer retries against a different deployment or a
higher-tier model. Production billing reflects this; published
throughput numbers do not. The unit $\CPMserved$ closes this gap by
counting only SLO-compliant tokens in the denominator.

Third, the unit is not free of the workload mix. The same system at the
same hardware can sit at different points on the frontier depending on
$(P, D)$ ratio, arrival rate, and SLO contract. This is the feature,
not the bug. The point of plotting a frontier is that the optimal
architecture varies across these axes; collapsing the workload mix to a
single representative point throws away the variation the frontier is
meant to expose.

We use $\CPMserved$ throughout. Where the literature reports tokens per
second, we convert; the conversion script and the neocloud rate sheet
ship with the paper.

% =========================================================================
\section{The architectures, cleanly stated}
\label{sec:architectures}

Three serving architectures matter in 2026. They differ in how prefill
and decode share GPU time and KV memory. The differences are
mechanical, not philosophical, and the cost frame in
Section~\ref{sec:decomposition} treats them mechanically.

\subsection{Colocated continuous batching}
\label{sec:cont-batch}

Orca \citep{yu2022orca} introduced iteration-level scheduling: at each
model forward pass, the scheduler chooses a batch from the pool of
in-flight requests, runs one iteration (one prefill chunk or one decode
step), and reschedules. vLLM's PagedAttention
\citep{kwon2023pagedattention} paged the KV cache into fixed-size
blocks so that fragmentation no longer caps realized batch size. The
two combine into the colocated continuous batching architecture that
has shipped in vLLM, SGLang \citep{zheng2024sglang}, TGI, and Triton.

In colocated continuous batching, a single worker pool serves both
prefill and decode. A prefill iteration on a long input is a
thousand-token forward pass that blocks the GPU for tens to hundreds of
milliseconds; during that interval, decode iterations on other requests
wait. TPOT inflates whenever a long prefill lands in the schedule. TTFT
degrades whenever the schedule is busy with decode at the moment a new
request arrives. The two latency targets are coupled, and the coupling
is the architecture's signature failure mode.

\subsection{Chunked-prefill colocation}
\label{sec:chunked}

Sarathi-Serve \citep{agrawal2024sarathi} splits each prefill into
chunks small enough that one prefill chunk plus a batch of decode steps
fits in a single forward pass. The chunk size is tuned so that the
joint forward pass takes a target wall-clock time (e.g.\ 30~ms),
bounding the TPOT inflation from prefill bursts. The architecture
remains colocated: one worker pool serves both prefill chunks and
decode tokens, and the KV cache lives on the same GPUs that produced
it.

Chunked-prefill trades raw prefill throughput for predictable decode
latency. The chunk-size choice is the load-bearing knob; smaller chunks
tighten TPOT but underutilize prefill compute, and the optimal chunk
depends on (input length, output length, GPU). Sarathi-Serve reports up
to $5.6\times$ higher serving capacity than vLLM on conversational
mixes at fixed SLO; the win shrinks on long-context inputs where the
chunks themselves dominate the forward pass.

\subsection{Prefill/decode disaggregation}
\label{sec:disagg}

DistServe \citep{zhong2024distserve}, Splitwise
\citep{patel2024splitwise}, and Mooncake \citep{qin2025mooncake} split
prefill and decode across two worker pools. Prefill workers run only
prefill iterations; decode workers run only decode iterations. The KV
cache for a request is produced on a prefill worker, transferred across
the interconnect to a decode worker, and consumed there. No prefill
iteration ever shares a GPU with a decode iteration. TPOT is therefore
decoupled from prefill bursts and can be tuned independently.

The disaggregation choice has two costs. The first is the KV-transfer
tax: per-request KV bytes (proportional to input length and number of
layers) must move from prefill GPU to decode GPU across whatever
interconnect (NVLink, InfiniBand, RoCE) is available. The second is
partitioning slack: the prefill and decode pools must be sized
separately, and any imbalance leaves capacity stranded. Splitwise
reports $1.4\times$ higher throughput at the same cost (equivalently,
${\sim}29\%$ lower cost at fixed throughput) on Azure's conversational
mix; Mooncake reports a $5\times$ throughput uplift at fixed cost on
Kimi-scale long-context conversational traffic where the prefill/decode
imbalance is most extreme.

\subsection{What we hold fixed}
\label{sec:held-fixed}

The frontier we plot isolates the architectural variable. Three levers
are held fixed across all three architectures and all SLO bins:

\begin{itemize}
\item \emph{Model.} Llama-2-70B \citep{touvron2023llama2} serves as the
  reference dense model in BF16 across A100~/~H100 plots; we callout
  FP8 sensitivity once in Section~\ref{sec:openproblems}. The Llama-2
  family was chosen because every cited serving paper either runs on
  Llama-2-70B directly or reports figures convertible to Llama-2-70B by
  the parameter-count and KV-layout corrections named in
  Section~\ref{sec:caveats}.
\item \emph{Decoding strategy.} Greedy decoding with batch size
  determined by the scheduler. Speculative decoding
  \citep{leviathan2023speculative} is held off because draft-target
  placement under disaggregation is its own scheduling problem
  (Section~\ref{sec:openproblems}).
\item \emph{Quantization.} BF16 weights, BF16 KV cache. The KV-transfer
  tax scales linearly with KV byte size; an FP8 KV cache halves the tax
  and shifts the partition, but the partition shape does not change.
\end{itemize}

Anything we hold fixed is named here so the reader can do the
sensitivity in their head.

% =========================================================================
\section{Decomposition}
\label{sec:decomposition}

\subsection{The total-cost equation}
\label{sec:total-cost}

For a serving system $S$ on hardware $H$ with model $M$ under workload
$D$ at arrival rate $\lambda$ and SLO contract, the cost per million
SLO-compliant served tokens decomposes as
\begin{equation}
\CPMserved \;=\; \alpha\,c_{\mathrm{prefill}}
  \;+\; (1{-}\alpha)\,c_{\mathrm{decode}}
  \;+\; \mathbb{1}_{\mathrm{P/D}}\,\tau_{\mathrm{KV}}
  \;+\; (1{-}\mathbb{1}_{\mathrm{P/D}})\,\sigma_{\mathrm{co}}.
\label{eq:decomp}
\end{equation}
The four terms are: $\alpha \in [0,1]$, the prefill compute share;
$c_{\mathrm{prefill}}$ and $c_{\mathrm{decode}}$, per-MTok costs of
each phase amortized over the realized batch in that phase;
$\tau_{\mathrm{KV}}$, the KV-transfer tax charged only in disaggregated
mode; and $\sigma_{\mathrm{co}}$, the colocation slack charged only in
colocated mode. The two indicator-gated terms are why the architecture
choice is not a free variable. Disaggregation buys you out of
$\sigma_{\mathrm{co}}$ and pays $\tau_{\mathrm{KV}}$; colocation does
the inverse trade. The break-even surface is the locus where the two
trades cost the same, which is what Section~\ref{sec:frontier} plots.

Three modeling choices keep the form tractable. We hold model $M$,
precision, and parallelism degree fixed across the three architectures
inside any one slice of the frontier; we hold the SLO contract fixed
inside any one figure; we treat $\alpha$ as workload-determined rather
than scheduler-determined. None of these is innocuous; each is named in
Section~\ref{sec:caveats} with the size of the error it introduces.

\subsection{The prefill share \texorpdfstring{$\alpha$}{alpha}}
\label{sec:alpha}

For a dense transformer with $n$ parameters, the FLOPs cost of
prefilling $P$ input tokens is ${\approx}\,2nP$ and the FLOPs cost of
decoding $D$ output tokens is ${\approx}\,2nD$ over the request
lifetime \citep{hoffmann2022chinchilla}. The prefill FLOPs share is
therefore
\begin{equation}
\alpha \;=\; \frac{P}{P + D},
\label{eq:alpha-def}
\end{equation}
independent of model size, at fixed precision and topology. The
Mooncake conversational+code trace summary reports a workload-wide
$\alpha$ near $0.85$ in production at Moonshot, dominated by
long-prompt summarization and document QA traffic
\citep{qin2025mooncake}. Re-deriving from the Splitwise paper's Azure
conversational-trace length statistics in \S3 of
\citet{patel2024splitwise} (mean input on the order of ${\sim}1{,}150$
tokens; mean output on the order of ${\sim}210$ tokens), the
workload-wide $\alpha = P/(P+D) \approx 0.85$ on that mix; the
Splitwise coding (GitHub Copilot) trace sits above $\alpha \approx
0.95$ on the same derivation, and we hold it out of the validation pass
in Section~\ref{sec:validation} because no published headline pins to a
single coding-trace operating point. Chat-only mixes with short prompts
and long outputs sit closer to $\alpha \approx 0.20$; summarization
mixes sit above $\alpha \approx 0.90$. The same deployment can
therefore sit at different points on the frontier as the workload mix
shifts during the day, and we return to this in
Section~\ref{sec:openproblems}.

Note that $\alpha$ is the FLOPs share, not the wall-clock share. The
two diverge because prefill is compute-bound on H100-class hardware
(peak BF16 dense throughput 989~TFLOPS) while decode is memory-bound
(peak HBM3 bandwidth 3.35~TB/s on H100 SXM5); both figures are taken
from the NVIDIA Hopper whitepaper \citep{nvidia2022hopper}. At batch
size one and BF16 weights, decoding Llama-2-70B requires loading
${\approx}\,140$~GB of weights per output token, capping single-stream
decode throughput at $3350/140 \approx 24$ tokens per second per H100.
Batching amortizes the weight read across multiple decode iterations
and lifts realized throughput; the realized decode batch enters
$c_{\mathrm{decode}}$ directly.

\subsection{The KV-transfer tax \texorpdfstring{$\tau_{\mathrm{KV}}$}{tau\_KV}}
\label{sec:tau-kv}

For a dense model with $L$ layers, $h_{kv}$ key/value heads after GQA,
head dimension $d$, and $b$ bytes per element, the KV bytes per token
per layer is $2 h_{kv} d b$. The KV bytes per request is
\begin{equation}
\KVreq \;=\; 2 L h_{kv} d b P.
\label{eq:kv-req}
\end{equation}
For Llama-2-70B with $L=80$, $h_{kv}=8$ (GQA-8), $d=128$, $b=2$ (BF16):
\begin{equation}
\KVreq \;=\; 2 \cdot 80 \cdot 8 \cdot 128 \cdot 2 \cdot P
  \;=\; 327{,}680\,P~\mathrm{bytes} \;=\; 320\,P~\mathrm{KB}.
\label{eq:kv-num}
\end{equation}
A 4096-token prompt produces ${\approx}\,1.31$~GB of KV bytes per
request; a 16{,}384-token prompt produces ${\approx}\,5.24$~GB.
Transferring these bytes from prefill GPU to decode GPU consumes
wall-clock time $t_{\mathrm{KV}} = \KVreq / B$, where $B$ is the
realized point-to-point bandwidth between the two pools. For NVLink~4
inside a node, peak $B \approx 900$~GB/s; for 400~Gbps RoCE across
pods, $B \approx 50$~GB/s; for HDR InfiniBand, $B \approx 25$~GB/s
\citep{nvidia2022hopper}. At the same 4K-token prompt, the same payload
transfers in ${\approx}\,1.5$~ms, ${\approx}\,26$~ms, and
${\approx}\,52$~ms respectively. The three regimes differ by a factor
of $36$ on the same prompt ($900 / 25$); this is why
$\tau_{\mathrm{KV}}$ is a first-class term and not a footnote.

The cost form amortizes the transfer time across served tokens:
\begin{equation}
\tau_{\mathrm{KV}} \;=\;
  \frac{c_H \cdot t_{\mathrm{KV}}}{(P + D) \cdot b_{\mathrm{dec}}} \cdot 10^6,
\label{eq:tau-kv}
\end{equation}
where $c_H$ is the hourly price of one accelerator and
$b_{\mathrm{dec}}$ is the realized decode batch sustained on the decode
pool. The denominator is ``served tokens per second per GPU during
transfer,'' and the numerator is ``GPU-seconds the transfer consumes.''
Two regimes follow. If the transfer pool keeps up with the decode
pool's demand for new KV slabs, $\tau_{\mathrm{KV}}$ is a per-token tax
that scales with KV bytes per token. If the transfer pool falls behind,
decode workers stall and the tax is paid as stranded decode capacity,
which inflates $\tau_{\mathrm{KV}}$ super-linearly in $P$ until the
pool sizing absorbs the imbalance. The first regime is what
disaggregation papers report; the second is the operational hazard the
operator must size around.

Mooncake's KV-transfer overlap optimization pipelines the transfer with
the first decode iterations on the destination worker: the paper
reports up to 75\% of $t_{\mathrm{KV}}$ can be hidden behind the first
$L_{\mathrm{overlap}}$ decode steps in production traffic on Kimi-scale
long-context workloads \citep{qin2025mooncake}. We carry this as a
single overlap factor $\eta \in [0,1]$ and write the validated tax as
$(1-\eta)\tau_{\mathrm{KV}}$;
Section~\ref{sec:validation-overlap} fits $\eta$ to the published
crossover.

\subsection{The colocation slack \texorpdfstring{$\sigma_{\mathrm{co}}$}{sigma\_co}}
\label{sec:sigma-co}

Colocated continuous batching incurs a slack term that disaggregation
does not: prefill iterations on long inputs stretch the TPOT
distribution of in-flight decode requests, and to keep $\TPOT_{99}$
inside the contract the scheduler must either limit the prefill batch
(underutilizing prefill compute) or trim the decode batch in any
forward pass that contains a prefill chunk (underutilizing decode
compute). Chunked-prefill colocation \citep{agrawal2024sarathi} reduces
$\sigma_{\mathrm{co}}$ by capping the prefill chunk size so the joint
forward pass fits a target wall-clock budget; the slack does not vanish
because the chunk-size choice itself trades prefill efficiency against
TPOT tail.

A first-order model treats $\sigma_{\mathrm{co}}$ as a function of
$\alpha$ and the chunk size $\chi$. Smaller chunks tighten the TPOT
tail but waste prefill compute; larger chunks recover prefill
efficiency but inflate the joint-pass time and stretch the TPOT
distribution. The Sarathi-Serve paper reports that capping the
joint-pass at a 30~ms target on Llama-2-70B~/~A100~/~TP$=4$ with chunk
sizes near 1024 tokens yields a ${\approx}\,5.6\times$
serving-capacity uplift over vLLM on conversational mixes at fixed
$(500~\mathrm{ms},\,50~\mathrm{ms})$ SLO contract
\citep{agrawal2024sarathi}. Reading the uplift back into
$\sigma_{\mathrm{co}}$ gives a chunked-prefill colocation slack of
${\approx}\,15\%$ utilization at that SLO bin on that workload (our
re-derivation, not a Sarathi-Serve-reported figure). Tightening
TPOT$_{99}$ from 50~ms to 30~ms costs a further ${\approx}\,20\%$ in
the Sarathi-Serve frontier on the same mix; this is the slack we charge
in Section~\ref{sec:frontier}'s tight-SLO slice.

\subsection{The break-even surface}
\label{sec:breakeven}

The colocated cost is
\begin{equation}
\CPM_{\mathrm{co}} \;=\;
  \alpha\,c_{\mathrm{prefill}}^{\mathrm{co}}
  \;+\; (1{-}\alpha)\,c_{\mathrm{decode}}^{\mathrm{co}}
  \;+\; \sigma_{\mathrm{co}}(\alpha,\,\chi,\,\mathrm{SLO}).
\label{eq:cpm-co}
\end{equation}
The disaggregated cost is
\begin{equation}
\CPM_{\mathrm{P/D}} \;=\;
  \alpha\,c_{\mathrm{prefill}}^{\mathrm{P/D}}
  \;+\; (1{-}\alpha)\,c_{\mathrm{decode}}^{\mathrm{P/D}}
  \;+\; (1{-}\eta)\,\tau_{\mathrm{KV}}(P,\,B).
\label{eq:cpm-pd}
\end{equation}
Disaggregation pays $(1{-}\eta)\tau_{\mathrm{KV}}$ and recovers
$\sigma_{\mathrm{co}}$. The break-even surface is the locus
\begin{equation}
\alpha\,(c_{\mathrm{prefill}}^{\mathrm{co}}
       - c_{\mathrm{prefill}}^{\mathrm{P/D}})
  \;+\; (1{-}\alpha)\,(c_{\mathrm{decode}}^{\mathrm{co}}
                       - c_{\mathrm{decode}}^{\mathrm{P/D}})
  \;+\; \sigma_{\mathrm{co}}
  \;=\; (1{-}\eta)\,\tau_{\mathrm{KV}}.
\label{eq:breakeven}
\end{equation}
Because disaggregation eliminates phase interference, the two
LHS-difference terms are non-negative whenever the SLO contract binds;
both are zero in the limit of infinite SLO slack and grow as the
contract tightens. The RHS $\tau_{\mathrm{KV}}$ is monotone in $P$ at
fixed $B$ and decreasing in $B$ at fixed $P$. Holding the SLO contract
fixed and varying $(\alpha,\,B)$ traces out the break-even surface in
workload-shape $\times$ interconnect-bandwidth space. The surface is
monotone: tighter contracts move the partition toward smaller $P$
(disaggregation wins on a broader range of inputs); higher bandwidth
moves the partition toward smaller $\alpha$ (disaggregation wins on a
broader range of mixes).

We solve the surface in closed form at the three SLO bins from the
topic memo, $(200~\mathrm{ms},\,30~\mathrm{ms})$,
$(500~\mathrm{ms},\,50~\mathrm{ms})$, and
$(1~\mathrm{s},\,80~\mathrm{ms})$, and plot it as Figure~\ref{fig:two}
in Section~\ref{sec:frontier}.

\subsection{Modeling choices and their bite}
\label{sec:caveats}

Five simplifications are load-bearing and worth naming. First, $\alpha$
as workload-determined treats the scheduler as a passive observer of
the workload mix; in production, schedulers admit or defer requests
and can shift the realized $\alpha$. The error this introduces is
bounded by the admission control band, which is at most $\pm 0.05$ on
the workloads we measure. Second, $c_{\mathrm{prefill}}^{\mathrm{co}}$
versus $c_{\mathrm{prefill}}^{\mathrm{P/D}}$ are treated as
architecture-only differences; in practice the prefill pool in P/D may
run at higher TP or different batch shapes than the colocated pool. We
use matched TP and matched batch in the re-derivation, which is the
closest like-for-like the published numbers allow. Third,
$\sigma_{\mathrm{co}}$ as a single scalar collapses a distribution; a
richer model would carry the TPOT tail explicitly. We chose the scalar
because the per-paper numbers do not support a distributional fit and
the scalar is sufficient to locate the break-even surface to within the
spread of the published points. Fourth, $\eta$ is a single overlap
factor; in reality, overlap quality depends on $L_{\mathrm{overlap}}$,
decode batch shape, and topology. We pin $\eta = 0.5$ as a conservative
default and report sensitivity in
Section~\ref{sec:validation-overlap}. Fifth, the DistServe anchor in
Figure~\ref{fig:one} is scaled from the paper's reported OPT-66B
configuration to Llama-2-70B by parameter count.
$c_{\mathrm{prefill}}$ is linear in $n$ to within architecture
corrections of $\pm 10\%$ (OPT and Llama-2 differ in layer count and
head dimension but share the standard $2nP$ FLOPs accounting), while
$c_{\mathrm{decode}}$ and $\KVreq$ are anchored to Llama-2's GQA-8
layout rather than OPT's vanilla MHA. The DistServe anchor on the
Pareto envelope carries a horizontal-axis error band of $\pm 10\%$ in
$\CPMserved$, smaller than the spread across SLO bins but worth
disclosing here so the \S\ref{sec:frontier-pareto} anchor placement is
not read as a 1{:}1 transfer of OPT-66B numbers to Llama-2-70B.

% =========================================================================
\section{The frontier}
\label{sec:frontier}

Three figures carry the empirical content. LaTeX \& Figures owns the
production of the plots from the calibration CSV and the
figure-generation script; this section describes what the plots show
and where each anchor point's numbers come from.

\subsection{Figure 1: cross-system Pareto frontier}
\label{sec:frontier-pareto}

\begin{figure}[t]
  \centering
  \includegraphics[width=0.92\linewidth]{figures/figure_1_pareto.pdf}
  \caption{Cross-system Pareto frontier on $(\TTFT_{99},\,\CPMserved)$
  at fixed $\TPOT_{99}=50$~ms, Llama-2-70B~/~H100~SXM5~/~TP$=4$. Five
  anchor points from \citet{kwon2023pagedattention,
  agrawal2024sarathi, zhong2024distserve, patel2024splitwise,
  qin2025mooncake}. The DistServe anchor is scaled from OPT-66B to
  Llama-2-70B by parameter count under the modeling choice of
  \S\ref{sec:caveats}, with a $\pm 10\%$ horizontal-axis error band.
  The shaded band sweeps the KV-transfer overlap factor $\eta \in
  [0.3,\,0.8]$ across the disaggregated anchors. Continuous-batching
  colocation sits inside the envelope on every slice. Source:
  \texttt{calibration.csv} + \texttt{anchors.csv} (pinned
  2026-05-01). Reproducibility:
  \texttt{wedge-ai-systems-eng/figures/generate\_figures.py}.}
  \label{fig:one}
\end{figure}

Figure~\ref{fig:one} plots $\CPMserved$ (vertical axis) against
$\TTFT_{99}$ (horizontal axis) at fixed $\TPOT_{99} = 50$~ms, on
Llama-2-70B~/~H100~SXM5~/~TP$=4$ deployments. Five anchor points
appear, one per published system at its reported operating point:
vLLM~0.6 with PagedAttention \citep{kwon2023pagedattention} on
continuous-batching colocation; Sarathi-Serve at its conversational and
coding configurations \citep{agrawal2024sarathi}; DistServe at the
OPT-66B configuration scaled by parameter count to Llama-2-70B under
the named modeling choice and $\pm 10\%$ horizontal-axis error band
disclosed in \S\ref{sec:caveats} \citep{zhong2024distserve}; Splitwise
at the Azure conversational-mix configuration
\citep{patel2024splitwise}; and Mooncake at the Moonshot
mixed-workload configuration \citep{qin2025mooncake}. Each point's
$\TTFT_{99}$ is taken directly from the cited paper's primary table;
each point's $\CPMserved$ is computed from the cited
tokens-per-second-per-GPU under SLO, scaled by CoreWeave's published
H100 SXM5 hourly rate as of 2026-05-01 \citep{coreweave2026pricing},
using the conversion in Section~\ref{sec:unit}. The conversion
arithmetic and the rate sheet ship with the figure as
\texttt{calibration.csv}.

The Pareto envelope through the five points partitions cleanly. Below
the break-even $\TTFT_{99}$ derived in Section~\ref{sec:breakeven},
every Pareto-optimal point is a P/D system: KV-transfer cost is
amortized over enough decode tokens to be cheaper than the colocation
slack at tight contracts. Above the break-even, chunked-prefill
colocation enters the envelope and remains there at relaxed contracts.
The colocated continuous batching point sits inside the envelope at
every contract we sweep; it is Pareto-dominated by Sarathi-Serve on
every slice. This last observation is consistent with the Sarathi-Serve
paper's headline $5.6\times$ finding \citep{agrawal2024sarathi}, but
the frontier carries it to its conclusion: there is no SLO contract on
Llama-2-70B~/~H100 at which continuous-batching colocation is the
cheapest delivered-token architecture.

\subsection{Figure 2: break-even surface}
\label{sec:frontier-be}

\begin{figure}[t]
  \centering
  \includegraphics[width=\linewidth]{figures/figure_2_breakeven.pdf}
  \caption{Break-even surface
  $\CPM_{\mathrm{co}}-\CPM_{\mathrm{P/D}}$ over $(\alpha,\,B)$ at the
  three SLO contracts $(200,\,30)$, $(500,\,50)$, $(1000,\,80)$~ms.
  Positive values (darker) mean disaggregation wins. The heavy contour
  is the crossover $\CPM_{\mathrm{co}}=\CPM_{\mathrm{P/D}}$.
  KV-transfer overlap is fixed at $\eta = 0.5$ across the panels.
  Source: \texttt{calibration.csv} (pinned 2026-05-01). Reproducibility:
  \texttt{wedge-ai-systems-eng/figures/generate\_figures.py}.}
  \label{fig:two}
\end{figure}

Figure~\ref{fig:two} is a heatmap of
$\CPM_{\mathrm{co}} - \CPM_{\mathrm{P/D}}$ over the plane
$(\alpha,\,B)$, with $\alpha$ on the horizontal axis sweeping from
$0.2$ (chat-heavy) to $0.97$
(long-context-summarization-heavy) and $B$ on the vertical axis
sweeping from $25$~GB/s (HDR InfiniBand) to $900$~GB/s (NVLink~4
intra-node). The crossover contour
$(\CPM_{\mathrm{co}} = \CPM_{\mathrm{P/D}})$ is overlaid. Three panels,
one per SLO bin, sit side-by-side for the three contracts:
$(200,\,30)$, $(500,\,50)$, $(1000,\,80)$.

The crossover contour moves with the SLO contract in the way
Section~\ref{sec:breakeven} predicts. At the tight contract
$(200,\,30)$ the contour is concave and sits at low $B$ across most of
the $\alpha$ range: disaggregation wins almost everywhere because the
colocation slack is large. At the relaxed contract $(1000,\,80)$ the
contour is convex and sits at high $B$ across most of the $\alpha$
range: disaggregation wins only on the prefill-heavy long-context
corner because the colocation slack is small and $\tau_{\mathrm{KV}}$
remains. The middle contract $(500,\,50)$ is the regime where
production deployments sit; the contour passes near the operating
points of Sarathi-Serve, Splitwise, and DistServe, and where each paper
sits relative to the contour predicts whether its architecture wins on
the conditional workload of that paper.

\subsection{Figure 3: hardware small multiples}
\label{sec:frontier-hw}

\begin{figure}[t]
  \centering
  \includegraphics[width=\linewidth]{figures/figure_3_hardware.pdf}
  \caption{Hardware small multiples. The Figure~\ref{fig:one} frontier
  swept across A100~SXM4, H100~SXM5, and H200~SXM5. Decode costs scale
  inversely with HBM bandwidth; prefill costs scale inversely with
  BF16 dense TFLOPS. The crossover marker is the TTFT at which the
  envelope flips from chunked-prefill to disaggregation. Hardware
  constants from \citet{nvidia2020ampere, nvidia2022hopper,
  nvidia2023h200}; rates from \citet{coreweave2026pricing}. Reproducibility:
  \texttt{wedge-ai-systems-eng/figures/generate\_figures.py}.}
  \label{fig:three}
\end{figure}

Figure~\ref{fig:three} holds Figure~\ref{fig:one} fixed and sweeps the
hardware: A100 SXM4, H100 SXM5, and H200 SXM5, three panels. A100 has
the smaller HBM2e bandwidth (2.04~TB/s) and lower BF16 dense
throughput (312~TFLOPS) \citep{nvidia2020ampere}; the crossover moves
to higher $\alpha$ because both
$c_{\mathrm{decode}}^{\mathrm{co}}$ and
$c_{\mathrm{decode}}^{\mathrm{P/D}}$ rise but the prefill-side gap
shrinks. H100 SXM5 sits at 3.35~TB/s HBM3 and 989~TFLOPS BF16 dense
\citep{nvidia2022hopper}. H200 has the larger HBM3e bandwidth
(4.80~TB/s) at the same 989~TFLOPS BF16 dense as H100
\citep{nvidia2023h200}; the crossover moves to lower $\alpha$ because
$c_{\mathrm{decode}}$ on both sides falls and $\tau_{\mathrm{KV}}$ is
unchanged (interconnects are the same), tilting the trade toward the
side that pays no transfer tax in the decode-light regimes and toward
the side that pays no slack in the prefill-light regimes. The
Blackwell extrapolation is held to Section~\ref{sec:openproblems}.

\subsection{Locating the published systems}
\label{sec:frontier-locate}

The four published systems, plus vLLM, occupy distinct regions of
Figure~\ref{fig:two}. Sarathi-Serve sits in the upper-left quadrant:
high SLO slack, decode-heavy and conversational-heavy mixes, low
$\alpha$. DistServe sits in the lower-right quadrant: tight SLO slack,
prefill-heavy mixes, high $\alpha$. Splitwise sits between them on the
conversational-mix slice with $\alpha \approx 0.85$ re-derived from
the trace statistics in \S\ref{sec:alpha}. Mooncake sits in the
upper-right corner of Figure~\ref{fig:two} (long-context, high
$\alpha$) and exploits the overlap factor $\eta$ to push the crossover
further toward the high-$\tau_{\mathrm{KV}}$ regime than the bare
decomposition predicts. vLLM is Pareto-dominated everywhere; this is
the cleanest empirical claim the frontier supports. Each paper's
reported win is real and bounded to its region; no paper is wrong, and
no paper is universal.

% =========================================================================
\section{Validation}
\label{sec:validation}

\subsection{Predicted vs.\ reported crossover}
\label{sec:validation-crossover}

The break-even surface from Section~\ref{sec:breakeven} makes three
quantitative predictions on cross-paper data:

\begin{enumerate}
\item DistServe vs.\ Sarathi-Serve on the Sarathi-Serve conversational
  mix at $(500,\,50)$: re-derived in the common cost frame from the
  operating points published in \citet{agrawal2024sarathi,
  zhong2024distserve} (\texttt{calibration.csv}), the predicted
  crossover sits at $\alpha \approx 0.45$ on H100~/~NVLink.
  Re-derived win regions: Sarathi-Serve dominates at $\alpha = 0.40$
  and DistServe dominates at $\alpha = 0.50$ on the same mix.
  Crossover within $\pm 0.05$; within the modeling band of
  Section~\ref{sec:caveats}. The win-region labels are the author's
  re-derivation in the common frame; neither source paper conducts the
  $\alpha$-binned head-to-head directly.

\item Splitwise vs.\ vLLM on Azure conversational mix at $(500,\,50)$:
  re-derived from the conversational-trace operating points in
  \citet{patel2024splitwise} (\texttt{calibration.csv}), the predicted
  throughput uplift at $\alpha = 0.85$ is ${\approx}\,1.45\times$ at
  the same cost (equivalently, ${\sim}31\%$ lower cost at fixed
  throughput). Splitwise's reported headline on the same trace is
  $1.4\times$ higher throughput at the same cost (equivalently,
  ${\sim}29\%$ lower cost at fixed throughput)
  \citep{patel2024splitwise}. Throughput-axis relative gap:
  $(1.45-1.40)/1.40 \approx 3.6\%$. Cost-axis relative gap:
  $(0.310-0.286)/0.286 \approx 8.4\%$. Both framings reproduce the
  predicted-vs-reported gap to within a few percent on the throughput
  axis and approximately two percentage points on the cost-reduction
  axis.

\item Mooncake vs.\ its colocated baseline on Kimi long-context
  conversational+code at $(1000,\,80)$: re-derived from the operating
  point in \citet{qin2025mooncake} (\texttt{calibration.csv}), the
  predicted throughput uplift at $\alpha = 0.85$, $B = 900$~GB/s,
  $\eta = 0.7$ is $4.3\times$. Reported: $5\times$ throughput uplift
  at fixed cost \citep{qin2025mooncake}. Within $14\%$; the residual
  is the gap Section~\ref{sec:validation-overlap} addresses. The
  $4.3\times$ figure is the author's re-derivation in the common
  frame, not a Mooncake-reported number.
\end{enumerate}

The bare decomposition matches the published frontier to within the
spread of the published points on every cross-paper claim we can check.
Where we cannot check, we say so: Llumnix \citep{sun2024llumnix}
reports an end-to-end speedup over DistServe and vLLM on a workload mix
we cannot reconstruct in enough detail to place on the surface, so we
hold it as an open data point.

\subsection{Where the model under-predicts: KV-transfer overlap}
\label{sec:validation-overlap}

The Mooncake residual in (\ref{sec:validation-crossover}.3) tracks one
omission: the bare decomposition charges the full
$\tau_{\mathrm{KV}}$ and Mooncake reports an upper bound of $75\%$
overlap on the most favorable Kimi long-context traces. We fit a
single $\eta = 0.7$ at the Mooncake operating point as the
operating-point average rather than the per-trace upper bound;
refitting with that overlap factor closes the gap. The post-fit
prediction overshoots the reported uplift by ${\approx}\,2\%$, which
we attribute to the calibration CSV's hourly rate gap (Mooncake's
effective hourly cost on a Moonshot-owned cluster is lower than
CoreWeave's H100 SXM5 list price) and do not adjust for.

The overlap factor is not free; it depends on $L_{\mathrm{overlap}}$
(number of decode steps the transfer overlaps with), the decode batch
shape, and the topology. We pin $\eta = 0.5$ as the default for
Figures~\ref{fig:one} and~\ref{fig:two}, $\eta = 0.7$ for the Mooncake
operating point only, and report Figure~\ref{fig:one} sensitivity to
$\eta \in [0.3,\,0.8]$ as a shaded band on the P/D frontier. The band
shifts the crossover by $\le 0.05$ in $\alpha$ on every slice; the
partition shape does not change.

\subsection{Where the model over-predicts: small-batch decode}
\label{sec:validation-small-batch}

The bare decomposition over-predicts the colocation slack
$\sigma_{\mathrm{co}}$ in the small-batch regime, where decode
utilization is already below the SLO knee and additional prefill
jitter does not bind. On the vLLM conversational baseline at $\lambda
\le 4$ requests per second per H100, the realized
$\sigma_{\mathrm{co}}$ is ${\approx}\,5\%$ rather than the
${\approx}\,15\%$ the chunk-size model predicts. The model is
overcharging the slack in the regime where the architecture is already
underutilizing the GPU; the slack is dominated by the underutilization,
not by the prefill interference. We carry this as a known calibration
gap and clip $\sigma_{\mathrm{co}}$ at the realized utilization in the
figure-generation script; the clip moves the colocated frontier inward
by 5--8\% at low $\lambda$ and does not change the partition.

\subsection{Re-profiling and re-derivation}
\label{sec:validation-reprofile}

The validation pass deliberately re-derives from published numbers
rather than re-profiling on private hardware. Dooly \citep{dooly2026}
is the current state of the art on configuration-agnostic,
redundancy-aware profiling for inference simulation; it reports within
$5\%$ MAPE on TTFT and $8\%$ on TPOT across two GPU platforms while
reducing profiling GPU-hours by $56\%$ \citep{dooly2026}. Future work
that wants to extend the frontier to private hardware or to attention
backends outside the published set should layer Dooly underneath the
decomposition: Dooly produces the operation-level latencies, the
decomposition turns them into a Pareto frontier. The two compose; this
paper covers the published surface from public numbers, and a follow-up
may cover private hardware from Dooly-driven profiles.

% =========================================================================
\section{Implications for production engineering}
\label{sec:implications}

\subsection{Pick the architecture from the frontier, not from the headline}
\label{sec:impl-frontier}

The five published systems in Section~\ref{sec:frontier} each report a
real win. None is wrong; each is bounded to a region of the frontier.
A production team facing an architecture decision should locate its
own $(\alpha,\,B,\,\mathrm{SLO})$ on Figure~\ref{fig:two} before
reading any paper's headline number. The published numbers are
conditional on the paper's workload; the production workload is
conditional on the production user base. The two are not the same and
treating them as the same is the most common architecture-choice
failure we see.

\subsection{KV-transfer bandwidth is a first-class procurement variable}
\label{sec:impl-bandwidth}

The break-even surface's sensitivity to $B$ is the single most
underappreciated finding in the public systems literature. Moving from
HDR InfiniBand to NVLink~4 shifts the partition by a factor of $36$ in
$t_{\mathrm{KV}}$ on the same prompt $(900 / 25)$, and the partition
shift is large enough to flip the architecture choice on coding and
summarization workloads. A disaggregated deployment behind 200~Gbps
RoCE will pay $\tau_{\mathrm{KV}}$ at the high end of the band and may
sit outside the disaggregation-wins region of Figure~\ref{fig:two}
even on workloads where the underlying paper claimed a win. Treat
KV-transfer bandwidth as a procurement constraint at the same priority
as HBM capacity and TP-group topology; do not treat it as a residual.

\subsection{SLO contracts belong on the rate card}
\label{sec:impl-slo}

A served-token bill without a $(\TTFT_{99},\,\TPOT_{99})$ tag is the
wrong unit. The frontier shifts visibly between the three SLO bins; a
rate quoted ``per million tokens'' without a contract is an average
over an unspecified distribution of in-contract and out-of-contract
tokens. Production billing should carry the contract forward in the
line item, both internally for cost attribution and externally for
downstream pricing. Field Note \#1 \citep{bhardwaj2026inferencestack}
argued the same point at the API layer; the same logic applies one
layer down at the serving layer.

\subsection{Mixed deployments live on the frontier too}
\label{sec:impl-hybrid}

Production deployments do not have to be pure colocated or pure
disaggregated. Hybrid configurations (chunked-prefill within
disaggregated decode pools; partial disaggregation under load;
KV-cache offload to a third tier as in the multi-tier KV memory line
\citep{ganjihal2026multitierkv}) sit between the two regimes on
Figure~\ref{fig:two}. The frontier framework gives them a place to
land: a hybrid configuration is a point on the surface with a
fractional $\mathbb{1}_{\mathrm{P/D}}$ and a reduced
$\tau_{\mathrm{KV}}$ proportional to the fraction of traffic routed
through the disaggregated path. The break-even surface generalizes;
the partition stays partitioned.

\subsection{SLO-aware autotuning closes the loop}
\label{sec:impl-autotune}

Once the architecture is chosen, the operating point inside that
architecture is a search problem: chunk size in colocated,
prefill/decode pool ratio in disaggregated, batch caps in both.
SLO-Guard \citep{lysenstoen2026sloguard} formalizes this as
crash-aware, budget-consistent autotuning under SLO constraints; the
autotuner runs over the same $(\alpha,\,B,\,\mathrm{SLO})$ inputs the
frontier consumes, which means the decomposition is the right cost
model for the autotuner to optimize against. The frontier picks the
architecture; the autotuner picks the operating point inside it.

% =========================================================================
\section{Open problems}
\label{sec:openproblems}

\subsection{Dynamic switching under shifting workload mix}
\label{sec:open-dynamic}

The decomposition assumes a single $\alpha$ for the slice of the
frontier being plotted. Production workloads shift $\alpha$ on diurnal
cycles (chat-heavy in the evening, coding-heavy during the working
day) and on bursts (model-evaluation traffic, marketing pushes). The
static frontier locates the optimal architecture for the time-averaged
$\alpha$; the dynamic problem of switching architectures under a
non-stationary $\alpha$ is open. Llumnix \citep{sun2024llumnix}
addresses dynamic scheduling within a fixed architecture; the
cross-architecture dynamic problem is unsolved in the published
literature.

\subsection{Speculative decoding and disaggregation}
\label{sec:open-spec}

Speculative decoding \citep{leviathan2023speculative} interacts
non-trivially with both architectures. Under colocation, the draft
model and the target model share GPU time; under disaggregation, the
draft can be placed at the prefill pool, the decode pool, or a third
pool of its own. The placement choice changes the realized
$c_{\mathrm{decode}}$ and introduces a new transfer tax for the
draft-target acceptance round. We held speculative decoding fixed in
Section~\ref{sec:held-fixed} because resolving the placement problem
is its own paper; this is Field Note \#6 territory.

\subsection{The committed-spend frontier}
\label{sec:open-commit}

The frontier we plot is on public hourly pricing. Reserved-capacity
and committed-spend contracts shift the per-accelerator price by a
meaningful margin observed in practice, depending on commit length and
provider; we do not anchor a specific discount range here because the
public pricing pages referenced \citep{coreweave2026pricing, aws2026p5}
do not document a single primary band, and any specific range belongs
in a separate pricing-survey paper. The reserved-capacity frontier is
not just a scaled version of the on-demand frontier: the partition
shifts because $c_{\mathrm{prefill}}$ and $c_{\mathrm{decode}}$ shift
by different multipliers under different commit structures
(commit-on-decode vs.\ commit-on-prefill is a real procurement
option), and $\tau_{\mathrm{KV}}$ does not scale with commit. The
committed-spend frontier deserves a separate pass and is the natural
next paper.

\subsection{Hybrid model architectures}
\label{sec:open-hybrid}

Mamba \citep{gu2023mamba}, Jamba \citep{lieber2024jamba}, and other
state-space~/~attention hybrids shift $\alpha$ and $\tau_{\mathrm{KV}}$
simultaneously: the state-space layers have constant-size state rather
than growing-with-context KV cache, which shrinks $\KVreq$ on long
inputs by orders of magnitude. On a pure state-space model,
$\tau_{\mathrm{KV}}$ collapses and disaggregation has no economic
argument left; on a hybrid, the partition redraws around the hybrid's
effective KV size. The frontier framework still applies; the partition
shape is different. A hybrid-architecture frontier is a separate
paper.

% =========================================================================
\section{Conclusion}
\label{sec:conclusion}

The serving-architecture question in 2026 is not which paper to read.
It is which region of the frontier the workload occupies. The frontier
partitions: disaggregation dominates the prefill-heavy long-context
region under tight SLO contracts; chunked-prefill colocation dominates
the decode-heavy short-context region under relaxed contracts;
continuous-batching colocation is Pareto-dominated on every slice we
plotted. The partition is computable from public numbers and the
closed-form decomposition in Section~\ref{sec:decomposition}. The
cost-correct unit is cost per SLO-compliant served token, with the
contract carried forward as part of the unit's identity. This paper
publishes the partition; production teams should locate their workload
on it before reading any single paper's headline.

\section{Cite this article}
\label{sec:bibtex}

\begin{verbatim}
@article{bhardwaj2026costfrontier,
  author = {Manu Bhardwaj},
  title  = {Disaggregated or Colocated?
            The Cost-Frontier of LLM Serving
            Under SLO Contracts},
  journal = {arXiv preprint},
  year   = {2026},
  url    = {https://ifitsmanu.com/papers/
             llm-serving-cost-frontier},
}
\end{verbatim}

% =========================================================================
\bibliography{references}

\papercolophon

\end{document}
