The Kotlin Benchmark is built to evaluate models on realistic, idiomatic Kotlin engineering tasks across multiple dimensions.
| Setup (Agent + LLM) proprietary | Submitted | Resolved | Resolution rate (%) | Avg. tokens (M) | Avg. latency | Date |
|---|---|---|---|---|---|---|
1Claude Code + Opus 4.7 xhigh | 105 | 90 | 85.71 | 10.59 | 10h 37m | 06.05.2026 |
2Junie (Opus 4.7 max) | 105 | 86 | 81.90 | 19.98 | 19h 58m | 23.04.2026 |
3Codex + GPT 5.5 xHigh | 105 | 86 | 81.90 | 38.78 | 8h 30m | 26.04.2026 |
4Claude Code + Opus 4.6 max | 105 | 84 | 80.00 | 13.93 | 12h 39m | 19.03.2026 |
5Codex + GPT 5.3 Codex xHigh | 105 | 82 | 78.10 | 8.56 | 8h 1m | 19.03.2026 |
6Junie (Opus-4.6) | 105 | 81 | 77.14 | 7.45 | 8h 36m | 23.03.2026 |
7Codex + GPT 5.4 xHigh | 105 | 81 | 77.14 | 10.85 | 9h 33m | 19.03.2026 |
8Claude Code + Opus 4.7 medium | 105 | 80 | 76.19 | 5.31 | 4h 54m | 07.05.2026 |
9Codex + GPT 5.4 medium | 105 | 80 | 76.19 | 8.29 | 6h 25m | 19.03.2026 |
10Claude Code + Sonnet 4.6 high | 105 | 78 | 74.29 | 12.17 | 13h 49m | 21.04.2026 |
11Claude Code + Opus 4.6 medium | 105 | 75 | 71.43 | 8.40 | 7h 43m | 18.03.2026 |
12Codex + GPT 5.3 Codex medium | 105 | 72 | 68.57 | 6.32 | 5h 26m | 19.03.2026 |
13Junie (GPT-5.4) | 105 | 72 | 68.57 | 15.29 | 11h 30m | 02.04.2026 |
14Claude Code + Sonnet 4.6 medium | 105 | 70 | 66.67 | 10.19 | 10h 28m | 19.03.2026 |
15Junie (GPT-5.3-codex) | 105 | 69 | 65.71 | 15.68 | 11h 8m | 23.03.2026 |
16Gemini CLI (Gemini 3.1 Pro) | 105 | 69 | 65.71 | 21.03 | 17h 1m | 21.03.2026 |
17Junie (Gemini 3 Flash) | 105 | 64 | 60.95 | 40.17 | 11h 27m | 23.03.2026 |
18Junie (Gemini 3.1 Pro Preview) | 105 | 63 | 60.00 | 32.41 | 13h 41m | 23.03.2026 |
19Gemini CLI (Gemini 3 Flash) | 105 | 47 | 44.76 | 36.53 | 11h 28m | 23.03.2026 |
See how the benchmark is built — the dataset of real Kotlin tasks, the harness that runs them, and how every result is scored.