Chunked-Prefill LLM

Fairness-Aware and Latency-Controllable Scheduling for Chunked-Prefill LLM Serving

arXiv:2606.09061v1 Announce Type: new Abstract: As large language models (LLMs) are increasingly deployed with highly heterogeneous workloads, chunked-prefill execution has emerged as a mainstream serving architecture. Balancing scheduling fairness and latency stability in such environments is critical; otherwise, severe head-of-line blocking and request starvation will degrade user experience. However, existing systems rely on rigid First-Come, First-Served (FCFS) policies and static token...