Brogue-CE Evaluation

Countless adventurers before you have descended this torch-lit staircase, seeking the promised riches below. As you reach the bottom and step into the wide cavern, the doors behind you seal with a powerful magic...

Welcome to the Dungeons of Doom!

Brogue is a roguelike game in which the player must descend through 26 levels of a dungeon to retrieve the Amulet of Yendor before returning to the surface. Within the dungeon there are deadly creatures and traps, but also weapons, potions, and powerful objects. Success requires making tactical decisions on a long time horizon: different actions and routes through the dungeon might be advantageous or treacherous, and every action matters.

We like Brogue as a task because it:

• requires complex interactions and strategy over long horizons
• can be played repeatedly, which allows for tracking in-session learning dynamics
• can be scored flexibly in ways that scale as capabilities improve
  • for example: once agents are able to beat the game, we can shift the metric to the number of moves required to beat the game (and compare against human speedrunning baselines)

Additionally, Brogue is very difficult. A new player might take dozens or hundreds of tries before completing a full descent and ascention. Experienced players sometimes only manage to win ~25% of the time. And Brogue, being a roguelike, has nice properties compared to other games (like Portal or RuneScape), such as game instances being generated from seeds (not memorizable), which allows for unlimited held-out tasks.

The maintained version of the game is open source on GitHub. We built a custom programmatic interface for playing the game, which removes reaction-time and pixel-recognition requirements (so that we can measure learning dynamics without real-time interaction or multimodality being confounders). Agents can interact with the game via an API similar to the one shown below:

{"request_id":"00000000-0000-4000-8000-000000000000","action":"move","direction":"ne"}
{"request_id":"00000000-0000-4000-8000-000000000000","action":"travel","x":42,"y":17}
{"request_id":"00000000-0000-4000-8000-000000000000","action":"wait"}
{"request_id":"00000000-0000-4000-8000-000000000000","action":"apply","item":"a"}
{"request_id":"00000000-0000-4000-8000-000000000000","action":"throw","item":"c","x":42,"y":17}
{"request_id":"00000000-0000-4000-8000-000000000000","action":"select","value":"a"}

We ran some initial experiments and found that the task has headroom when evaluated on GPT-6 Astra at maximum reasoning. We also implemented a number of soundness and hardening checks in development.

We detail the experiments and hardening below.

Experiments

The experimental setup is inspired by EBR-Bench. We initially give the agent 10 playthroughs of the game. The first 8 are for learning and exploration, and are not scored. The last 2 are for performance and we take the better run as the final score. Each playthrough uses a distinct, unknown dungeon, initialized with a fixed seed.

Later, we experiment with giving the agent 30 playthroughs in a session, with a 12 hour total time limit. This allows us to get a better picture of the learning dynamic over many interactions.

Agents are able to take and persist notes throughout games, and are run in a single session, with context compaction enabled.

We measure progress on a 0–52 scale: descending earns up to 25 points, acquiring the Amulet earns 26, and returning toward the surface while carrying it earns up to 51. Escaping alive earns 52. Before acquiring the Amulet, progress is the deepest credited floor minus one.

Astra at medium reasoning (v3)

We ran 10 independent sessions of GPT-6 Astra at medium reasoning, each with eight learning games and two scored games. Each session used its own set of ten dungeon seeds and started with a fresh conversation and workspace. Here we report the eight completed 8+2 sessions, excluding the two sessions that were interrupted.

The mean final score—the mean of each completed session's better scored game—was 5.9/52. No game acquired the Amulet or escaped. Bars show mean progress at each game index; the dashed line shows the mean running best.

Bars show the mean and pointwise 95% bootstrap intervals across eight sessions. The line averages each session's best so far, including learning games. Only games 9 and 10 are scored.

Astra at maximum reasoning (v5)

We then ran one complete 8+2 session of GPT-6 Astra at maximum reasoning, using the corrected interface. Both scored games reached floor 7, giving a final score of 6/52. The best learning games reached floor 8. No game acquired the Amulet or escaped.

Each bar is one game's earned progress; the dashed line is the running best. Games 9 and 10 are scored.

Thirty-game sessions: Astra medium and max (v6)

We next ran one 30-game session at medium reasoning and one at maximum reasoning. All 30 games counted, and the highest progress reached in any game was the final score. There was no prescribed learning phase: the agent chose when to explore and when to exploit. The sessions used the same fresh schedule of 30 distinct, initially unknown dungeons, with separate conversations and persistent workspaces.

Neither session acquired the Amulet or escaped. Medium first reached its best score in game 24 and matched it in game 29. Max reached its best in game 11; its best subsequent game reached floor 8. Max used the entire time allowance; medium finished with about 4 h 36 min remaining.

This is not a perfect experiment -- it's possible that the agents would improve over more than 30 games. Also, in the max run, the agent sometimes ended games early to stay within the time limit. We are interested in scaling up experiments, but are sharing these initial ones to give a sense for what we're interested in.

Soundness

Task soundness is something that we care a lot about. In development, we found a number of potential soundness issues, which we mitigate in a number of ways. Here are some examples:

• we locked native difficulty mode controls (so agents cannot set the mode to "easy")
• grading via programmatic state that the agent cannot access
• we fixed information leakage via the game's "hallucination" mechanic, which initially was implemented in a way that provided the agent with extra information

Between v3 and v6 -- the current version -- we also fixed or clarified three interface issues:

• throwing an equipped or enchanted item could dismiss the game's confirmation prompt without actually throwing it
• overlapping client commands could be mistaken for the same request or rejected because another request was pending. The client now processes individual game commands one at a time
• the agent guide described a 100×34 display without explaining that text rows omit trailing spaces, which could cause indexing errors in agent-written parsers. We clarified that agents should pad text rows before indexing or use the complete rendered grid

Updated 2026-09-10 00:44 UTC