tokenspeed

TokenSpeed is a high-performance LLM inference engine designed for agent workloads. It combines the performance of TensorRT-LLM with the usability of vLLM, aiming to become the most efficient inference engine in production environments. The core components of TokenSpeed include the modeling layer, scheduler, kernel system, and entry point. The modeling layer adopts a local SPMD design, generating collective communication via a static compiler; the scheduler uses a C++ control plane and a Python execution plane; the kernel system is pluggable with a centralized registry; and the entry point integrates SMG to enable low-overhead CPU-side request handling.