Kimi K3 is an open-weights model that outperforms closed systems from GPT and Anthropic on several independent benchmarks. Its weights are publicly available for local deployment, yet the model leads on key metrics despite the typical resource disadvantages faced by open projects.
Developers scaled three architectural dimensions at once: context length reaching one million tokens, increased layer depth, and a Mixture of Experts design with 896 specialists. Each direction introduced distinct engineering challenges that were solved without sacrificing training stability or inference efficiency.
Handling One Million Tokens with a Fixed-Size State Matrix
Standard transformers rely on a growing KV-cache whose memory footprint scales linearly with sequence length. For a million tokens this cache would exhaust available GPU memory. Kimi K3 replaces the KV-cache with a fixed-size state matrix S updated by the delta-rule. Two learned scalars control the update: beta governs how strongly new information overwrites old values, while alpha controls forgetting. To prevent underflow and overflow in BF16 arithmetic during long sequences, alpha is clamped between e^-5 and 1.
Parallelizing Sequential State Updates
The delta-rule update is inherently sequential, conflicting with GPU parallelism. The team introduced the open-source FlashKDA kernel built on NVIDIA CUTLASS. Inside each chip, tokens within a chunk are processed in parallel before state hand-off occurs across attention heads. An automatic scheduler further distributes long-context prefill across multiple streaming multiprocessors on a single card. Across cards, the computation is split into local state increments and transition matrices; an all-gather followed by prefix-scan reconstructs the exact sequential states with communication volume independent of context length.
Preserving Information Across Deep Layers
Deeper networks risk losing early-layer features as activations propagate through many transformations. Each layer in Kimi K3 maintains a learned pseudo-query that retrieves a weighted combination of outputs from all preceding blocks rather than receiving only the immediate predecessor. Layers are grouped into eight blocks of twelve layers each; intra-block connections remain standard while inter-block selection uses the pseudo-query mechanism, dramatically reducing memory required to store prior activations.
Scaling to 896 Experts with Stable Training
The model employs two always-active shared experts operating on full-dimensional tokens and 896 routed experts that receive compressed latent representations. For every token only 16 routed experts are selected. To prevent activation explosion across four consecutive matrix multiplications, the team inserted normalization before one multiplication and replaced SwiGLU with the custom SiTU-GLU activation whose branches are mathematically bounded. Load balancing is achieved by per-expert quantile thresholds that guarantee each expert receives a predefined share of tokens; these thresholds are computed efficiently via distributed histograms rather than full sorting. Experts are never split across GPUs, and a small per-GPU reserve ensures every card receives exactly its planned token volume.
