LLM Inference Introduction: the end-to-end flow and vector math
Every vector an LLM computes during inference, from token ids to logits, with the shapes and the code.
Every vector an LLM computes during inference, from token ids to logits, with the shapes and the code.
What the KV cache costs per token, and how MQA, GQA and MLA rewrite attention to make that number smaller.
FlashInfer treats LLM attention as a serving systems problem spanning data layout, kernel generation, and runtime scheduling.
A walkthrough of numeric formats that matter today (FP32, TF32, FP16, BF16, FP8, FP4, INT8, INT4), how they behave, when to use each, and where the pitfalls hide.
The scheduler is the layer between paged memory and fast kernels, and it decides whether either of them matters.
What quantization actually changes in a network, which parts of a transformer break first, and which techniques repair them.