<aside> 📢
<aside> 🎯
</aside>
<aside>
In auto-regressive Large Language Model (LLM) generation, execution is fundamentally split into two phases: a compute-heavy prefill phase and a memory-bandwidth-bound decoding phase. During the prefill phase, the entire prompt is processed in parallel with self-attention operating at O(L²) complexity, resulting in high arithmetic intensity and making this stage primarily compute-bound.
During decoding, the model generates one token at a time, requiring the inference engine to repeatedly read massive weight matrices from High Bandwidth Memory (HBM). Because the number of FLOPs performed per byte transferred is exceedingly low, the GPU cannot fully utilize its Tensor Cores, making the decoding stage strictly memory-bandwidth-bound. To mitigate this memory wall and improve hardware utilization, standard inference engines employ techniques that either reduce memory traffic or increase the useful work done per memory fetch:
⚠️ Limitations & Caveats
$$
\begin{aligned} & \text{[Arithmetic Intensity]: } I = \frac{\text{Total FLOPs}}{\text{Memory Bytes Transferred}} \\ & \text{[KV Cache Size per Token]: } S_{token} = 2 \times n_{layers} \times n_{heads} \times d_{head} \times \text{precision bytes} \\ & \text{[Total KV Cache Size]: } S_{total} = B \times L \times S_{token} \quad (B = \text{Batch}, L = \text{Seq Len}) \\ & \text{[Symmetric Quantization]: } X_{int} = \text{round}\left(\frac{X_{float}}{s}\right), \quad s = \frac{\max(|X_{float}|)}{2^{b-1}-1} \\ & \text{[Dequantization]: } \tilde{X}{float} = X{int} \times s \end{aligned} $$
The KV Cache stores the Key and Value tensors produced by each self-attention layer for previously generated tokens. When the model predicts the token at position t, it computes the Query, Key, and Value only for that specific position. The new Query is then compared against the cached Keys and Values from positions 1 to t−1 to compute attention scores, reducing the time complexity of the decoding step.
In early inference implementations, memory for the KV cache was allocated as large contiguous blocks sized for the maximum possible sequence length. Because real user requests have highly variable and unpredictable lengths, this static allocation strategy leads to significant memory fragmentation.
This fragmentation manifests in two forms: Internal fragmentation occurs when a sequence terminates long before reaching the reserved maximum length, leaving unused memory trapped within the allocated block. External fragmentation occurs when free VRAM becomes divided into many small non-contiguous regions that cannot satisfy a large contiguous allocation request for a newly arriving sequence.
⚠️ Limitations & Caveats
Standard static batching processes multiple sequences synchronously. Because all sequences must advance in lockstep, shorter requests idle while waiting for the longest sequence in the batch to finish, creating GPU utilization bubbles. Continuous batching (iteration-level scheduling) improves utilization by dynamically inserting new requests into the batch the moment another request finishes generation.
To handle the highly dynamic memory allocation requirements of continuous batching, frameworks like vLLM (Kwon et al., 2023, arXiv:2309.06180) address the issue using PagedAttention. PagedAttention divides the KV cache into fixed-size memory blocks (pages) that do not need to be physically contiguous in VRAM. Each sequence maintains a logical mapping to its blocks via a block table, analogous to how operating systems manage virtual memory.
| Dimension | Standard Static Batching | Continuous Batching + PagedAttention |
|---|---|---|
| Computational Cost | Low scheduling overhead | Higher scheduling overhead |
| Data Requirements | Efficient only when prompt lengths are uniform | Efficient across highly diverse prompt lengths |
| Memory Efficiency | High waste due to contiguous pre-allocation | Near-zero waste (<4% internal fragmentation) |
| Known Failure Modes | GPU idle bubbles, frequent OOM crashes | Block table lookup overhead, hardware lock-in |
⚠️ Limitations & Caveats
<aside>
The development of a modern Large Language Model operates through a three-stage pipeline, transitioning a raw statistical engine into a steerable assistant. The foundational stage, Pretraining, uses self-supervised next-token prediction over internet-scale corpora. This computationally massive phase encodes grammar, world knowledge, and reasoning capabilities into the weights, but yields a base model that simply continues text distributions rather than following instructions.
To mold this base model into an assistant, practitioners apply Supervised Fine-Tuning (SFT). Using high-quality, human-curated instruction-response pairs (e.g., Ouyang et al., 2022, arXiv:2203.02155), SFT aligns the model's output to a conversational persona and specific formatting constraints. The final stage, Alignment, tunes the model to maximize helpfulness and harmlessness, traditionally requiring Reinforcement Learning from Human Feedback (RLHF). Direct Preference Optimization (DPO) simplifies this final phase by mathematically bypassing the reinforcement learning step entirely (Rafailov et al., 2023, arXiv:2305.18290).
⚠️ Limitations & Caveats
$$
\begin{aligned} & \text{[Pretraining / SFT Loss]: } \mathcal{L}{SFT} = -\mathbb{E}{(x,y) \sim \mathcal{D}}[\log \pi_\theta(y|x)] \\ & \text{[Bradley-Terry Preference]: } P(y_w \succ y_l | x) = \frac{\exp(r(x, y_w))}{\exp(r(x, y_w)) + \exp(r(x, y_l))} \\ & \text{[RLHF Objective]: } \max_{\pi_\theta} \mathbb{E}{x \sim \mathcal{D}, y \sim \pi\theta}[r(x,y) - \beta \mathbb{D}{KL}(\pi\theta(y|x) || \pi_{ref}(y|x))] \\ & \text{[Optimal Policy Mapping]: } \pi^*(y|x) = \frac{1}{Z(x)} \pi_{ref}(y|x) \exp\left(\frac{1}{\beta} r(x,y)\right) \\ & \text{[DPO Loss Function]: } \mathcal{L}{DPO} = -\mathbb{E}{(x, y_w, y_l)} \left[ \log \sigma \left( \beta \log \frac{\pi_\theta(y_w|x)}{\pi_{ref}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{ref}(y_l|x)} \right) \right] \end{aligned} $$
In standard RLHF, practitioners train a separate Reward Model to output a scalar score based on the Bradley-Terry preference framework. The policy model is then optimized using algorithms like Proximal Policy Optimization (PPO) to maximize this reward, constrained by a KL-divergence penalty to prevent it from deviating too far from the reference model. This requires constantly sampling new outputs from the policy during training and passing them through the reward network.
DPO's mathematical insight is recognizing that the closed-form optimal solution to this specific RL objective can be algebraically inverted. By manipulating the equations, the unknown reward function is expressed entirely in terms of the log-ratio between the active policy probabilities (π_θ) and the frozen reference model (π_ref), scaled by the KL penalty parameter β.
By substituting this reparameterized reward back into the preference model, the explicit reward function mathematically cancels out. The alignment objective is thereby transformed into a maximum likelihood estimation problem where the model simply increases the log-probability of the chosen response and decreases the log-probability of the rejected response.
⚠️ Limitations & Caveats
Traditional RLHF using PPO requires loading four distinct neural networks into memory simultaneously: the active Policy Model, the frozen Reference Model, the Reward Model, and the Value Network (Critic). This massive VRAM requirement forces heavy reliance on tensor parallelism and multi-node orchestration. Furthermore, PPO generates text dynamically during the training loop, which drastically increases the time-per-step and introduces high variance.
DPO reduces this architectural footprint by requiring only two models to be loaded: the active Policy Model and the frozen Reference Model. It operates strictly on offline data, meaning no text generation occurs during the forward pass. This allows DPO to utilize standard gradient descent over batches, making it highly stable and accessible to smaller engineering teams, though often at the cost of peak alignment performance on complex reasoning tasks.
| Dimension | Standard RLHF (PPO) | Direct Preference Optimization (DPO) |
|---|---|---|
| Computational Cost | High (~4 models in VRAM, active text generation) | Moderate (~2 models in VRAM, no generation step) |
| Data Requirements | Preference data for Reward Model, plus diverse, unlabelled prompts | Large, static dataset of explicitly paired chosen/rejected responses |
| Known Failure Modes | Reward hacking, mode collapse, extreme hyperparameter sensitivity | Overfitting to static dataset, weaker out-of-distribution generalization |
⚠️ Limitations & Caveats
<aside> <img src="/icons/reorder_gray.svg" alt="/icons/reorder_gray.svg" width="40px" />
</aside>