Batching & Scheduling Optimization in LLM Inference

Discrete-Event Simulation of LLM Serving Systems

2025.09 - 2025.12

Overview

A discrete-event simulation (DES) framework for rigorously analyzing the performance trade-offs between system throughput, tail latency, and fairness in LLM serving — replicating the behavioral physics of state-of-the-art systems like vLLM and Orca.

Team: Jiedong Zhang, Qingyang Xu, Runyuan He.

Motivation

LLM inference is a stochastic service system with highly variable service times, two-phase execution (compute-bound prefill vs. memory-bound decoding), and stringent latency requirements. Traditional M/M/1 queuing models are inadequate for such heterogeneous, stateful workloads — batching and scheduling decisions need rigorous simulation-based analysis.

Key Findings

1. Batch Size Optimization

Batch size has a non-monotonic relationship with system performance:

B ≤ 8: severe throughput bottleneck — at B=1, average latency exceeds 870s with a queue explosion
B = 16–32: optimal operating point — near-maximum throughput (9.9 req/s) while maintaining low latency (~2.3–2.7s)
B ≥ 64: no further gain — the "Memory Wall" caps decoding throughput

2. Scheduling Policies & Fairness

With a bimodal workload (70% short / 30% long requests) at near-saturation:

SJF cuts average latency 37% vs. FCFS (9.15s vs 14.65s), but P99 explodes to 190s, Jain's Fairness Index drops from 0.87 to 0.09, and 7% of requests starve
Priority with aging is the balanced compromise: fairness 0.84, latency −6%

3. Load Stress & Saturation

Saturation occurs around 25 req/s (throughput caps at 20 req/s). Recommended operating rate: 20 req/s (80% of saturation) to maintain SLO guarantees.

4. Adaptive Batching under Traffic Spikes

Under spike load (25 req/s), the adaptive strategy (B=32 → B=64) reduces aggregate latency by 54% and max queue depth by 84% compared to static batching.

Methodology