#1 — Qwen-3.6-32B replicated on LiveCodeBench (+4.2 pts vs vendor card)
Full Replication
Independent evaluation across three community harnesses confirmed Qwen-3.6-32B-Instruct achieves 68.4% pass@1 on LiveCodeBench (2026-Q3 split) in BF16. Two separate reproductions matched within 0.3 percentage points using the standard v0.5 evaluation container.
Validates that the 32B dense checkpoint outperforms 70B-class predecessors on contamination-resistant coding tasks on a single 80GB GPU. Teams running self-hosted code generation can drop two-GPU tensor parallelism without losing pass@1 accuracy.
Gap: Vendor card used greedy decoding without temperature=0.2 pass@1 harness flag.
#2 — Llama-4-Scout-17B MoE drops 6.8 pts on SWE-bench Verified under INT4 AWQ
Partial Replication
Third-party harness testing on SWE-bench Verified shows Llama-4-Scout-17B-16E drops from 54.1% (BF16) to 47.3% when quantized with 4-bit AWQ group-size 128. Router gate precision degradation accounts for the majority of failed multi-file patch diffs.
Operators deploying MoE checkpoints in 4-bit to fit consumer or L40S GPUs should keep router and shared-expert layers in FP16. Mixed-precision quantization recovers 5.9 of the lost 6.8 points at only +1.4 GB VRAM overhead.
Gap: Tested on 300-instance stratified subset; full 500-instance v1.2 harness run pending.
"Llama-4-Scout-17B-16E scores 54.1% resolved in BF16 but falls to 47.3% (-6.8 pts) under standard INT4-AWQ g128."
#3 — DeepSeek-R2-Lite-14B claims 81.2% on AIME 2026 at 12k max reasoning tokens
Vendor Claim
The model card for DeepSeek-R2-Lite-14B reports 81.2% pass@1 on AIME 2026 with a 12,288 reasoning token budget, compared to 74.0% at 32k tokens on the prior R1-Distill-14B checkpoint. Weights are released under MIT license on Hugging Face.
If independently replicated, cutting reasoning chain length by 62% directly reduces time-to-first-answer and KV-cache memory pressure in batch math/verification pipelines. Treat as unverified until community harnesses confirm token-budget enforcement.
Gap: Self-reported in release model card; evaluation seed and majority-voting exclusion scripts not yet pushed to repo.
"DeepSeek-R2-Lite-14B reaches 81.2% pass@1 on AIME 2026 under a strict 12,288 thinking-token cap."