SWE-bench Verified

In the last 30 days, SWE-bench Verified appeared in 1 passing signal (1 replicated, 0 vendor-reported). Latest (2026-10-25): Llama-4-Scout-17B MoE drops 6.8 pts on SWE-bench Verified under INT4 AWQ

• 2026-10-25 (#2): Llama-4-Scout-17B MoE drops 6.8 pts on SWE-bench Verified under INT4 AWQ — Third-party harness testing on SWE-bench Verified shows Llama-4-Scout-17B-16E drops from 54.1% (BF16) to 47.3% when quantized with 4-bit AWQ group-size 128. Router gate precision degradation accounts for the majority of failed multi-file patch diffs. Partial Replication