THE EVIDENCE / CODING
Coding
benchmarks.
Software engineering and code generation tasks. Results are grouped by benchmark and unit. Different tests are never merged into a raw leaderboard.
01 / RESULTS
Published results
| Benchmark | Version | Method | Configuration | Model | Result | Test date | Verified | Source type | Source |
|---|---|---|---|---|---|---|---|---|---|
| CursorBench v3.2 | CursorBench v3.2 | Publisher-reported configuration; see source | — | grok-4-6 | 69.9 % | 2026-08-12 | 2026-09-29 | provider first party | Source ↗ |
| CursorBench v3.2 | CursorBench v3.2 | Publisher-reported configuration; see source | — | grok-4-5 | 66.7 % | 2026-08-12 | 2026-09-29 | provider first party | Source ↗ |
| DeepSWE v1.1 | DeepSWE v1.1 | Publisher-reported configuration; see source | — | grok-4-6 | 65.9 % | 2026-08-12 | 2026-09-29 | provider first party | Source ↗ |
| DeepSWE v1.1 | DeepSWE v1.1 | Publisher-reported configuration; see source | — | grok-4-5 | 54 % | 2026-08-12 | 2026-09-29 | provider first party | Source ↗ |
| FrontierCode v1.1 Extended | FrontierCode v1.1 Extended | Publisher-reported configuration; see source | — | grok-4-6 | 61.3 % | 2026-08-12 | 2026-09-29 | provider first party | Source ↗ |
| FrontierCode v1.1 Extended | FrontierCode v1.1 Extended | Publisher-reported configuration; see source | — | grok-4-5 | 56.6 % | 2026-08-12 | 2026-09-29 | provider first party | Source ↗ |
| Terminal-Bench 4.0 (reported best effort) | Terminal-Bench 4.0 (reported best effort) | Publisher-reported configuration; see source | — | claude-sonnet-5-5 | 70.6 % | 2026-09-28 | 2026-09-29 | provider first party | Source ↗ |
| Terminal-Bench 4.0 (reported best effort) | Terminal-Bench 4.0 (reported best effort) | Publisher-reported configuration; see source | — | claude-opus-5-5 | 66.4 % | 2026-09-28 | 2026-09-29 | provider first party | Source ↗ |
| CursorBench 4.0 (reported best effort) | CursorBench 4.0 (reported best effort) | Publisher-reported configuration; see source | — | claude-sonnet-5-5 | 55.5 % | 2026-09-28 | 2026-09-29 | provider first party | Source ↗ |
| CursorBench 4.0 (reported best effort) | CursorBench 4.0 (reported best effort) | Publisher-reported configuration; see source | — | claude-opus-5-5 | 57.8 % | 2026-09-28 | 2026-09-29 | provider first party | Source ↗ |
| FrontierCode 1.1 Main (reported best effort) | FrontierCode 1.1 Main (reported best effort) | Publisher-reported configuration; see source | — | claude-sonnet-5-5 | 46.2 % | 2026-09-28 | 2026-09-29 | provider first party | Source ↗ |
| FrontierCode 1.1 Main (reported best effort) | FrontierCode 1.1 Main (reported best effort) | Publisher-reported configuration; see source | — | claude-opus-5-5 | 54.4 % | 2026-09-28 | 2026-09-29 | provider first party | Source ↗ |
| Terminal-Bench 4.0 Vals mini-SWE-agent avg@3 fallbacks failed | 4.0 | Vals mini-SWE-agent avg@3; Anthropic fallbacks treated as failures | — | claude-opus-5-5 | 58.08 % | 2026-09-29 | 2026-09-29 | independent lab | Source ↗ |
| Terminal-Bench 4.0 Vals mini-SWE-agent avg@3 fallbacks failed | 4.0 | Vals mini-SWE-agent avg@3; Anthropic fallbacks treated as failures | — | claude-sonnet-5-5 | 62.63 % | 2026-09-29 | 2026-09-29 | independent lab | Source ↗ |
| Terminal-Bench 4.0 Vals mini-SWE-agent avg@3 fallbacks failed | 4.0 | Vals mini-SWE-agent avg@3; Anthropic fallbacks treated as failures | — | claude-fable-5-1 | 50 % | 2026-09-29 | 2026-09-29 | independent lab | Source ↗ |
| Terminal-Bench 4.0 Vals mini-SWE-agent avg@3 fallbacks failed | 4.0 | Vals mini-SWE-agent avg@3; Anthropic fallbacks treated as failures | — | gpt-6-astra | 59.6 % | 2026-09-29 | 2026-09-29 | independent lab | Source ↗ |
| ProgramBench Vals 200 public tasks raw pass rate | Vals 2026-09-27 | Vals 200 public tasks; raw pass rate; mini-SWE-agent | — | claude-opus-5-5 | 87 % | 2026-09-27 | 2026-09-29 | independent lab | Source ↗ |
| ProgramBench Vals 200 public tasks raw pass rate | Vals 2026-09-27 | Vals 200 public tasks; raw pass rate; mini-SWE-agent | — | gpt-6-astra | 85.4 % | 2026-09-27 | 2026-09-29 | independent lab | Source ↗ |
| ProgramBench Vals 200 public tasks raw pass rate | Vals 2026-09-27 | Vals 200 public tasks; raw pass rate; mini-SWE-agent | — | claude-fable-5-1 | 82.7 % | 2026-09-27 | 2026-09-29 | independent lab | Source ↗ |
| ProgramBench Vals 200 public tasks raw pass rate | Vals 2026-09-27 | Vals 200 public tasks; raw pass rate; mini-SWE-agent | — | gpt-6-sol | 81.9 % | 2026-09-27 | 2026-09-29 | independent lab | Source ↗ |
| LiveBench code_completion | 2026-06-25 | Public table; published model effort variant | gemini-3.1-pro-preview-high | gemini-3-1-pro-preview | 78.261 % | not published | 2026-09-30 | benchmark maintainer | Source ↗ |
| LiveBench code_generation | 2026-06-25 | Public table; published model effort variant | gemini-3.1-pro-preview-high | gemini-3-1-pro-preview | 74.648 % | not published | 2026-09-30 | benchmark maintainer | Source ↗ |
| LiveBench code_completion | 2026-06-25 | Public table; published model effort variant | gemini-3.5-flash-high | gemini-3-5-flash | 76.087 % | not published | 2026-09-30 | benchmark maintainer | Source ↗ |
| LiveBench code_generation | 2026-06-25 | Public table; published model effort variant | gemini-3.5-flash-high | gemini-3-5-flash | 80.282 % | not published | 2026-09-30 | benchmark maintainer | Source ↗ |
| LiveBench code_completion | 2026-06-25 | Public table; published model effort variant | grok-4.5 | grok-4-5 | 69.565 % | not published | 2026-09-30 | benchmark maintainer | Source ↗ |
| LiveBench code_generation | 2026-06-25 | Public table; published model effort variant | grok-4.5 | grok-4-5 | 67.606 % | not published | 2026-09-30 | benchmark maintainer | Source ↗ |
| LiveBench code_completion | 2026-06-25 | Public table; published model effort variant | gpt-5.6-sol-max | gpt-5.6-sol | 84.783 % | not published | 2026-09-30 | benchmark maintainer | Source ↗ |
| LiveBench code_generation | 2026-06-25 | Public table; published model effort variant | gpt-5.6-sol-max | gpt-5.6-sol | 83.099 % | not published | 2026-09-30 | benchmark maintainer | Source ↗ |
| LiveBench code_completion | 2026-06-25 | Public table; published model effort variant | gpt-5.6-terra-max | gpt-5.6-terra | 80.435 % | not published | 2026-09-30 | benchmark maintainer | Source ↗ |
| LiveBench code_generation | 2026-06-25 | Public table; published model effort variant | gpt-5.6-terra-max | gpt-5.6-terra | 76.056 % | not published | 2026-09-30 | benchmark maintainer | Source ↗ |
| LiveBench code_completion | 2026-06-25 | Public table; published model effort variant | gpt-5.6-luna-max | gpt-5.6-luna | 86.957 % | not published | 2026-09-30 | benchmark maintainer | Source ↗ |
| LiveBench code_generation | 2026-06-25 | Public table; published model effort variant | gpt-5.6-luna-max | gpt-5.6-luna | 78.873 % | not published | 2026-09-30 | benchmark maintainer | Source ↗ |
| LiveBench code_completion | 2026-06-25 | Public table; published model effort variant | gemini-3.5-flash-lite-high | gemini-3-5-flash-lite | 76.087 % | not published | 2026-09-30 | benchmark maintainer | Source ↗ |
| LiveBench code_generation | 2026-06-25 | Public table; published model effort variant | gemini-3.5-flash-lite-high | gemini-3-5-flash-lite | 76.056 % | not published | 2026-09-30 | benchmark maintainer | Source ↗ |
| LiveBench code_completion | 2026-06-25 | Public table; published model effort variant | gemini-3.6-flash-high | gemini-3-6-flash | 78.261 % | not published | 2026-09-30 | benchmark maintainer | Source ↗ |
| LiveBench code_generation | 2026-06-25 | Public table; published model effort variant | gemini-3.6-flash-high | gemini-3-6-flash | 77.465 % | not published | 2026-09-30 | benchmark maintainer | Source ↗ |
| LiveBench code_completion | 2026-06-25 | Public table; published model effort variant | deepseek-v4-pro-0813 | deepseek-v4-pro | 78.261 % | not published | 2026-09-30 | benchmark maintainer | Source ↗ |
| LiveBench code_generation | 2026-06-25 | Public table; published model effort variant | deepseek-v4-pro-0813 | deepseek-v4-pro | 76.056 % | not published | 2026-09-30 | benchmark maintainer | Source ↗ |
| LiveBench code_completion | 2026-06-25 | Public table; published model effort variant | grok-4.6 | grok-4-6 | 76.087 % | not published | 2026-09-30 | benchmark maintainer | Source ↗ |
| LiveBench code_generation | 2026-06-25 | Public table; published model effort variant | grok-4.6 | grok-4-6 | 77.465 % | not published | 2026-09-30 | benchmark maintainer | Source ↗ |
| LiveBench code_completion | 2026-06-25 | Public table; published model effort variant | gemini-3.7-flash-high | gemini-3-7-flash | 76.087 % | not published | 2026-09-30 | benchmark maintainer | Source ↗ |
| LiveBench code_generation | 2026-06-25 | Public table; published model effort variant | gemini-3.7-flash-high | gemini-3-7-flash | 81.69 % | not published | 2026-09-30 | benchmark maintainer | Source ↗ |
| LiveBench code_completion | 2026-06-25 | Public table; published model effort variant | claude-fable-5-1-max-effort | claude-fable-5-1 | 82.609 % | not published | 2026-09-30 | benchmark maintainer | Source ↗ |
| LiveBench code_generation | 2026-06-25 | Public table; published model effort variant | claude-fable-5-1-max-effort | claude-fable-5-1 | 90.141 % | not published | 2026-09-30 | benchmark maintainer | Source ↗ |
| LiveBench code_completion | 2026-06-25 | Public table; published model effort variant | gemini-3.8-flash-high | gemini-3-8-flash | 71.739 % | not published | 2026-09-30 | benchmark maintainer | Source ↗ |
| LiveBench code_generation | 2026-06-25 | Public table; published model effort variant | gemini-3.8-flash-high | gemini-3-8-flash | 73.239 % | not published | 2026-09-30 | benchmark maintainer | Source ↗ |
| LiveBench code_completion | 2026-06-25 | Public table; published model effort variant | gpt-6-astra-max | gpt-6-astra | 80.435 % | not published | 2026-09-30 | benchmark maintainer | Source ↗ |
| LiveBench code_generation | 2026-06-25 | Public table; published model effort variant | gpt-6-astra-max | gpt-6-astra | 80.282 % | not published | 2026-09-30 | benchmark maintainer | Source ↗ |
| LiveBench code_completion | 2026-06-25 | Public table; published model effort variant | deepseek-v4.1-flash-max | deepseek-v4-1-flash | 82.609 % | not published | 2026-09-30 | benchmark maintainer | Source ↗ |
| LiveBench code_generation | 2026-06-25 | Public table; published model effort variant | deepseek-v4.1-flash-max | deepseek-v4-1-flash | 77.465 % | not published | 2026-09-30 | benchmark maintainer | Source ↗ |
| LiveBench code_completion | 2026-06-25 | Public table; published model effort variant | claude-opus-5-5-max-effort | claude-opus-5-5 | 86.957 % | not published | 2026-09-30 | benchmark maintainer | Source ↗ |
| LiveBench code_generation | 2026-06-25 | Public table; published model effort variant | claude-opus-5-5-max-effort | claude-opus-5-5 | 91.549 % | not published | 2026-09-30 | benchmark maintainer | Source ↗ |
| LiveBench code_completion | 2026-06-25 | Public table; published model effort variant | gpt-6-sol-max | gpt-6-sol | 80.435 % | not published | 2026-09-30 | benchmark maintainer | Source ↗ |
| LiveBench code_generation | 2026-06-25 | Public table; published model effort variant | gpt-6-sol-max | gpt-6-sol | 83.099 % | not published | 2026-09-30 | benchmark maintainer | Source ↗ |
| LiveBench code_completion | 2026-06-25 | Public table; published model effort variant | gpt-6-luna-max | gpt-6-luna | 80.435 % | not published | 2026-09-30 | benchmark maintainer | Source ↗ |
| LiveBench code_generation | 2026-06-25 | Public table; published model effort variant | gpt-6-luna-max | gpt-6-luna | 77.465 % | not published | 2026-09-30 | benchmark maintainer | Source ↗ |
| LiveBench code_completion | 2026-06-25 | Public table; published model effort variant | claude-sonnet-5-5-max-effort | claude-sonnet-5-5 | 86.957 % | not published | 2026-09-30 | benchmark maintainer | Source ↗ |
| LiveBench code_generation | 2026-06-25 | Public table; published model effort variant | claude-sonnet-5-5-max-effort | claude-sonnet-5-5 | 95.775 % | not published | 2026-09-30 | benchmark maintainer | Source ↗ |
Limitations: Repository context, tool access and pass criteria vary across tests. Publication methods, prompts, sampling and model versions may differ. Check each cited source before interpreting a result.