A benchmark suite in Doom
Embodied cooperative multi-agent reinforcement learning
Cooperative MARL is almost always tested on state vectors or top-down 2D images. Embodied agents get neither. They see a private first-person camera, teammates vanish behind walls, and coordination has to come from what each agent can see.
COMRAD is a cooperative benchmark built on ViZDoom for exactly that setting. Every scenario is a compiled WAD, agents act through factorized discrete actions, and a standard baseline trains for 100M steps in about an hour on a single GPU.
13 purely cooperative scenarios spanning social dilemmas, role asymmetry, spatial and temporal synchronization, and combat.
DoomGen, a relaxed-Voronoi map generator that turns each scenario into a seedable distribution of layouts.
Two independent difficulty axes, task mechanics and spatial layout, with curriculum learning tools on top.
Seven CTDE algorithms (eight configurations) on one asynchronous Sample Factory stack: IPPO, MAPPO, HAPPO, IDQN, VDN, QMIX, QPLEX.Benchmarks are central to the development and evaluation of multi-agent reinforcement learning (MARL) algorithms. As the cooperative MARL community has grown, two categories of evaluation environments have proven indispensable: low-dimensional feature-vector benchmarks that isolate algorithmic behavior in compact state spaces, and two-dimensional pixel-based benchmarks that rely on overhead visual observations. Embodied cooperation, however, requires high-dimensional 3D egocentric perception, partial observability arising from line-of-sight occlusion, and the ability to close the gap between recognizing a coordination pattern and executing it from a first-person view. While various 3D environments have been explored for vision-based RL, no existing platform simultaneously provides a standardized, purely cooperative multi-agent benchmark with 3D first-person observations and high-throughput simulation.
To address this gap, we introduce COMRAD, a COoperative Multi-Agent Reinforcement LeArning benchmark suite in Doom, featuring a diverse set of challenging scenarios spanning role asymmetry, temporal synchronization, and spatial navigation. To introduce within-scenario variability, we develop DoomGen, a procedural map generator that produces diverse layout configurations for every scenario. We integrate COMRAD with Sample Factory, a high-throughput asynchronous RL framework, and implement seven MARL baselines on top of it, reaching ~25K frames per second during training. Our experiments show that COMRAD poses significant challenges for current CTDE methods, establishing visual cooperative MARL as an important open frontier.
| Benchmark | 3D ego | Vision | Co-op | Proc gen | CTDE base. | Open source | Agents |
|---|
Each agent sees a private 128×72 RGB camera, nothing else. Choose a scenario to see two in-game views, what it tests, and how the seven baselines scored after 100M steps on held-out maps.
A small browser remake of COMRAD's easiest scenario, with your partner being a bot, Rambo. Hares are worth a point and anyone can take them. The stag is worth five, but its shield only drops while you and your partner stand close together.
Hares, +1. Safe and solo. Greedy play racks these up and never learns the bigger payoff, the classic hare-versus-stag trap.
Stag, +5. Shielded unless both hunters are within range of each other. Coordination has to emerge from what each of you can see.
Hide the partner cam to play the way a decentralized policy acts: only your own view, no messages, no map.Agents trained on one map memorize where to meet and shoot. DoomGen builds playable WADs from a relaxed Voronoi tessellation, so geometry, sightlines and corridor lengths change while the objective, roles and rewards stay fixed.
ARMORY SIEGE · LAYOUT 1 / 6Same template, six seeds
Three supply rooms, one defended armory, timed doors at every bottleneck. DoomGen keeps that structure and redraws everything else.
Scenarios can also be trained on a finite pool of pre-generated WADs sampled per episode, which keeps every run exactly reproducible.
Task mechanics (enemy count, timing, hazards) and spatial layout (arena size, corridor length, occluders) are set independently. A method that survives one axis and fails the other tells you which skill is missing.
Armory Siege, one QMIX checkpoint, held-out maps
Select a cell. Spatial difficulty is the steeper axis: Easy to Hard costs 1.40 points on average, against 0.93 for mechanics.
Seven baseline configurations, 100M steps, teams of two, zero-shot on held-out layouts. Mean over five seeds (42, 68, 81, 97, 154), 100 evaluation episodes each.
Hover a row. Click a column header to follow one algorithm.The best on-policy method leads in 8 of 13 scenarios. The family split explains less than the scope of credit assignment.
Everything is shared: observations, action spaces, budgets. So the gaps in the scoreboard trace back to specific algorithmic mechanisms.
MAPPO beats IPPO on Rhythm Sync (0.67 vs 0.57) and Co-op Puzzle (3.82 vs 2.98). On role-asymmetric Ammo Carrier, IPPO scores more than five times higher (7031 vs 1351).
Next: critics whose inputs adapt to role or team context.
QMIX leads Foraging Commons and Armory Siege and is the only method to score on Stealth Labyrinth. Stale teammate trajectories and monotonic mixing break narrow synchronization windows.
Next: policy-age correction, teammate memory, set or graph critics.
HAPPO goes negative on Armory Siege (−0.85) and QPLEX-D sits near zero on five tasks. Their extra machinery is hard to train through convolutional encoders.
Next: event-level credit (handoffs, supply loops) and grounded communication.
Ammo Carrier pins one agent to the hub as a defender and sends the other on depot-to-hub supply runs. Trajectory density from each method's best checkpoint:
With decentralized updates the runner optimizes its loop on its own. Under centralized updates its gradients are shaped by the defender's value signal; the runner drifts off the route and the defender runs dry.
Every baseline scores near zero on Stealth Labyrinth and Lava Maze. Two pairs of human players, who could not talk or see each other's screens, show the tasks are learnable. Scripted partners show each role is learnable too; what fails is co-adaptation between roles.
ViZDoom is a CPU-bound C++ engine, so COMRAD pairs asynchronous CPU rollouts with GPU learning on Sample Factory instead of a JAX pipeline.
Team size vs throughput · Armory Siege, QMIX, one A100
Going from 2 to 8 agents cuts environment steps per second by 6.9× and quadruples engine sync timeouts. Within 30M frames only N=2 learns meaningful coordination.
Curriculum protocols · Platform Chain, MAPPO, 125M steps
PLR reaches a strong checkpoint at 19.1M steps. Sequential progression peaks highest (6.94); OMNI keeps the best final checkpoint (5.38).
@inproceedings{nguyen2026comrad,
title={{COMRAD}: A Benchmark for Embodied Multi-Agent Reinforcement Learning},
author={Khoi H.B. Nguyen and Dimitar Zhekov and Tristan Tomilin},
booktitle={Nineteenth European Workshop on Reinforcement Learning},
year={2026},
url={https://openreview.net/forum?id=0x7G2k0Hh7}
}