See how the Kotlin Benchmark is built: 105 tasks from eight open-source Kotlin repositories, SWE-bench-based methodology, and future plans.
The Kotlin Benchmark is based on the open-source Multi-SWE-bench harness. We extend Multi-SWE-bench, which already supports Java, Go, TypeScript, Rust, and other languages, with first-class Kotlin support:
The Kotlin Benchmark contains 105 tasks across eight repositories, drawn from open production Kotlin codebases. We selected repositories based on popularity (number of stars) and contributor activity. Each task is a real, merged GitHub pull request that implements a fix. The model is given the issue or PR description and must produce a patch that passes a hidden regression test.
Based on the Multi-SWE-bench infrastructure, the benchmark is adapted for Kotlin projects. Each task is mined from a merged pull request in a real Kotlin repository. For every task, we capture the base commit, the human-written “gold” solution, the regression tests that define the expected behavior, and the natural-language issue description. Every task builds a reproducible Docker image with a layered-cache strategy, making repeated runs faster while keeping evaluation deterministic. The agent receives the issue and repository state and must produce a patch that completes the task. A task is counted as resolved only when the generated solution passes the required tests.
We treat benchmarks as a continuous quality measurement pipeline. To improve coverage and usefulness, we’re already working on three areas:
We’re moving toward a data split that is more representative of the Kotlin ecosystem. Our current work focuses on expanding the task set into common Kotlin domains, such as Android and KMP, and introducing a range of difficulty levels so the benchmark better reflects real-world use cases.
Today, task verification uses tests as the primary correctness gate. In upcoming iterations, we’ll add non-functional dimensions, such as cost, token consumption, code quality, runtime performance, maintainability, and security. This will help you evaluate the best cost/intelligence trade-offs for your domain.
We plan to broaden the catalog of evaluated models to include more open-weight and open-source solutions.