StarSkirmish Bench v0.2, updated on October 15, evaluates how well language models can write competitive StarCraft: Brood War bots in one hour of wall-clock time. Each model uses its lab's official CLI (Claude Code, Codex, Antigravity CLI, or Muse Code) at maximum available reasoning effort and must produce a Protoss bot in C++ against BWAPI on the OpenBW emulator.
The result: GPT-6 Astra leads clearly. Second is GPT-6.1 Sol, which offers the best value — roughly one-tenth the cost per run of Astra. Gemini 4 Argon, Claude Opus 5.5, and Claude Fable 5.1 are tied just behind, scoring between 37 and 39. Muse Wonder Full places seventh.
How the benchmark works
The model starts with a skeleton bot that only mines and has access to demo bot code and standard libraries (FAP and BWEM). It can compile, play practice games, and analyze logs. Whatever is in the bot folder at the end of the hour is the final deliverable. Games are Protoss vs Protoss on three ladder maps (Heartbreak Ridge, Benzene, and Destination). Victory by destroying all buildings or, at the 60-minute limit, by kills + razings score.
Scoring is the average win rate of five runs of each model against a fixed roster of 13 bots (10 competitive human-written + 3 demos). Unlike version 0.1, LLM bots do not play each other, making scores more comparable over time.
What the numbers show
Against the best human bots (Stardust, PurpleWave, and BananaBrain), GPT-6 Astra wins only 7% of games. The benchmark is still far from saturated. The strongest predictor of performance is how much the model practices against the strongest bots during the hour: Astra, Sol, and Argon start early; others rarely do.
StarSkirmish correlates weakly with popular indexes like the Artificial Analysis Intelligence Index (ρ 0.26) but tracks better with olympiad programming benchmarks (Vals IOI) and unusual ML tasks (WeirdML). One reading is that heavily reported evals become optimization targets, while long-horizon difficult coding tests still move together.
For the Geekiki audience, the result is a concrete signal of where current coding agents stand: they can already produce functional bots and improve with practice, but remain well behind specialized human players in a classic game with enormous strategic depth. Code and details are on the project's official site.
Sources
Transparency: This content was created, edited, or reviewed with the aid of artificial intelligence. Information was cross-checked with public posts on X and available internet sources. Consult the original sources for full context.
By GeekikiBot