Kimi K3's technical report: what Moonshot disclosed and what it didn't
· TexaHub · LLMs & Foundation Models, Artificial Intelligence
Moonshot AI's 2.8-trillion-parameter open-weight model comes with a detailed report on architecture and post-training, an unusual government cyber assessment, and conspicuous gaps on data and compute.
The release
Moonshot AI announced Kimi K3 in mid-July and published its weights on Hugging Face on 27 July, together with a technical report on GitHub and parts of its infrastructure, including attention kernels and an MoE communication library [5]. The model card describes a Mixture-of-Experts model with 2.8 trillion total parameters, 104 billion activated, 93 layers and a context length of 1,048,576 tokens, and calls it "the world's first open 3T-class model" [2].
Openness here comes with conditions. The Kimi K3 License is permissive in most respects, but any licensee running a "Model as a Service" business with revenue above 20 million US dollars over 12 months must sign a separate agreement with Moonshot before commercial use, and products with more than 100 million monthly active users or 20 million US dollars in monthly revenue must display "Kimi K3" prominently [3]. For most researchers and enterprises running the model internally this changes little; for large inference providers it is a real constraint.
Architecture: three bets at once
The report describes three main changes from Kimi K2. First, attention is hybrid: 69 layers use Kimi Delta Attention (KDA), a linear-attention design for long-sequence mixing, interleaved with 24 Gated MLA layers that preserve global token-to-token attention [1][2]. The MLA layers use no explicit positional encoding (NoPE), with position information carried by KDA's gating and decay, which the authors say lets the model extrapolate to 1-million-token contexts without RoPE rescaling or interpolation [1].
Second, Attention Residuals let each layer attend to representations from all preceding layers [1]. Third, a Stable LatentMoE design routes each token to 16 of 896 experts, up from 8 of 384 in Kimi K2, with SiTU-GLU activations and a Quantile Balancing scheme to stabilise training at that sparsity [1]. The report's comparison table puts K2 at 1.04 trillion total and 32.6 billion activated parameters, against 2.78 trillion and 104.2 billion for K3 [1]. Context was extended in four stages, from 8K to 64K tokens in pre-training and from 256K to 1M tokens during cooldown [1].
Moonshot's headline claim is that these changes, with refined data and training recipes, yield an approximately 2.5-times improvement in overall scaling efficiency over K2 [1]. The figure comes from fitted scaling-law curves of validation loss against compute on held-out out-of-distribution data [1]. That is a meaningful internal measure, but it is the company's own fit, and the report does not let outsiders reproduce it.
What the report leaves out
The pre-training section lists the data domains (web text, code, mathematics and knowledge, plus a large vision corpus) and the filtering and rephrasing methods, but in the version we reviewed it does not state the number of training tokens or describe the training cluster [1]. For comparison, the Kimi K2 paper stated that model was pre-trained on 15.5 trillion tokens [7]. Without token counts or compute figures, the 2.5x efficiency claim cannot be checked against public scaling results, and questions about how much compute a 300-person company, as The Decoder described Moonshot, was able to marshal remain open [6].
Post-training is described in more detail: supervised fine-tuning, reinforcement learning across coding, general-agent and reasoning domains at several reasoning-effort levels, then consolidation into one model through multi-teacher on-policy distillation [1]. Quantization-aware training with MXFP4 weights and MXFP8 activations starts at the SFT stage, aimed at lowering serving memory and cost [1].
Benchmarks and an outside cyber test
By Moonshot's own account, K3 "still trails the most powerful proprietary models, namely Claude Fable 5 and GPT-5.6 Sol" overall, while outperforming the other models in its suite [1]. Its figures show K3 at 88.3 on Terminal-Bench 2.1 against 88.8 for GPT-5.6 Sol, and at 67.5 on DeepSWE against 73.0 for GPT-5.6 Sol and 70.0 for Claude Fable 5 [1]. The report cites third-party results as of 23 July, including an Artificial Analysis Intelligence Index of 57.1 behind Claude Fable 5 at 59.9 and GPT-5.6 Sol at 58.9 [1]. All of these are vendor-selected comparisons.
The most unusual element is an independent cyber evaluation. The UK AI Security Institute and the US Center for AI Standards and Innovation published a joint preliminary assessment on 23 July [4]. They found K3 scored 32% on ExploitBench against 24% for GLM-5.2, but achieved arbitrary code execution on "0/41 samples, whereas the most cyber-capable models achieved ACE on 20/41 samples on average" [4]. On a 32-step simulated attack path, K3 reached step 17 on average, against 28.5 for the most capable US models [4]. The institutes concluded that K3 "performs significantly below the most recent frontier cyber-capable models" but also that its "safeguards did not prevent it from attempting cyber exploit development" [4]. They noted caveats, including a selective set of evaluations and wider confidence intervals for K3 [4].
Why it matters
Commentary after the July announcement focused on whether a small team could reach near-frontier results with limited compute; SemiAnalysis founder Dylan Patel was quoted saying strong research in RL, architecture and data "helps make up for lot of the compute deficit" [6]. K3 is also no longer alone at this scale: on 3 August Alibaba launched Qwen3.8-Max, a 2.4-trillion-parameter MoE with 95 billion active parameters, with weights "scheduled for release next week" [8].
Our reading is that K3's report is strongest where it documents engineering: hybrid linear attention at trillion-parameter scale, extreme expert sparsity and low-precision training are now described in public detail. Its weakest point is the absence of data and compute disclosure, which is exactly what policymakers and competitors most want to know. The joint government assessment is a precedent worth watching, as it suggests open-weight releases from Chinese labs may increasingly be tested by Western institutes within days of launch.
Sources
[1] Kimi K3: Open Frontier Intelligence (technical report) — Moonshot AI, 28 Jul 2026
[2] moonshotai/Kimi-K3 model card — Moonshot AI, 27 Jul 2026
[3] Kimi K3 License — Moonshot AI, 27 Jul 2026
[4] Preliminary assessment of Kimi K3's cyber capabilities — UK AI Security Institute, 23 Jul 2026
[5] Moonshot AI releases Kimi K3 open weights and infrastructure after shaking up the frontier model race — The Decoder, 27 Jul 2026
[6] Just like Deepseek, China's Kimi K3 is forcing Western AI labs to question their compute advantage — The Decoder, 17 Jul 2026
[7] Kimi K2: Open Agentic Intelligence — arXiv, 28 Jul 2025
[8] Alibaba Unveils Qwen3.8-Max: Its Largest and Most Capable Flagship Model to Date — Alibaba Cloud, 3 Aug 2026