LLM safety · multi-agent RL · self-play
Self-play against only the latest opponent cycles and forgets. Population-based self-play (PSRO) has been proposed for red-teaming before; we ask what it takes to make it work for LLMs, and what breaks once the population is strong. The resulting defender is the safest among the self-play methods we compare.
Payoff matrix, attacker's payoff (A0, D0 = base model)
Nash defender after D5 joins
Schematic: each dot is one generation's defender, placed by its safety and helpfulness on held-out benchmarks. Up and to the right is better.
Motivation
In jailbreak self-play, an attacker writes prompts and a defender must answer safely. The space of possible attacks is open-ended, and no training set, benchmark or judge covers more than a small part of it. So any single training run settles on a strategy that fits the part it happened to see: a valid solution, but only one of many.
We trained the same attacker six times. Same recipe, same data; only the order of training samples changed. Each run still ended up with different winning strategies: the two strongest break 81 prompts each, yet share only 38. Training once is like drawing one ticket: you get one of many valid outcomes, and which one is luck. We call this partial convergence.
Rather than deploying k models, we want one policy that captures this coverage by training against the population.
One might expect a policy trained for longer to pick up all of these strategies by itself. Reinforcement learning works against this: it piles probability onto whichever strategy pays most right now. We trained one attacker against a fixed defender. As it trains, the entropy over the attack strategies it uses falls to 0.02× its starting value.
Token-level entropy, the usual diversity monitor, rises to 1.56× over the same steps. The attacker writes longer, more varied sentences around fewer ideas, so the monitor reports the opposite of what is happening. Training longer does not fix this.
Naive self-play trains each new defender against the newest attacker only, so nothing asks it to keep resisting earlier attacks. On WildJailbreak, the fourth defender is broken by 10% of prompts; the fifth, trained against a newer attacker, by 74%.
Strategies beat each other in cycles, so the defender is pushed around a loop instead of up a ladder: it swings between refusing broadly, which is safe but unhelpful, and complying broadly, which is helpful but unsafe.
Method
We build on Policy-Space Response Oracles (PSRO; Lanctot et al., 2017) and its use for red-teaming language models (GRTS; Ma et al., 2023). What we add is a study of what makes it work at 8B scale: which ingredient matters, why each member must start fresh, how to keep the payoff matrix affordable, and how a strong population exposes flaws in the reward.
The matrix grows quadratically and every cell needs thousands of judged conversations. A racing evaluation budget and a rollout cache cut that cost by about an order of magnitude.
Results
Every method starts from an abliterated Llama-3.1-8B-Instruct, whose safety training has been removed. PSRO ends up the safest, at a moderate helpfulness cost.
| Method | Safety ↑ | Helpful ↑ |
|---|---|---|
| Starting point | ||
| Abliterated Llama-3.1-8B-Instruct | .356 | .936 |
| Other safety training | ||
| Circuit Breakers | .579 | .916 |
| MAGIC | .850 | .920 |
| Self-play | ||
| Naive self-play | .638 | .886 |
| Self-RedTeam | .808 | .830 |
| Fictitious self-play | .890 | .844 |
| PSRO (ours) | .955 | .811 |
Safety averages seven harmful-prompt measures; Helpful averages benign compliance on WildJailbreak and XSTest. All trained rows start from the abliterated model. PSRO, fictitious self-play and MAGIC average two seeds.
Attack and defense strategies beat each other in cycles, like rock-paper-scissors. A model that only chases the newest opponent keeps trading one weakness for another instead of making progress.
A newer defender can lose to an attack an older one stopped. So the best practice mix often leans on older opponents. Here the newest attacker was not picked at all.
A defender trained against a well-chosen mix of past attackers also holds up against attackers trained after it. The strongest of them does worse every round.
Measuring how a new member does against everyone takes minutes; training it takes hours. Weighting all opponents equally skips that measurement, wastes practice on opponents already beaten, trains longer, and still ends up less safe.
A model that has already been trained hard has little room left to learn. Continuing it falls behind and then breaks into blank answers. Starting each member from the original model keeps every round trainable.
Once attackers got good, they found a loophole: instead of rewording the request they were given, they swapped in a harmful one, because the scoring paid for that. The population exposed a flaw we had not noticed.
After we fixed that loophole, a milder trick remained: swapping a hard request for an easier one. As self-play gets stronger, better scoring becomes the main thing to improve.
A model can learn to please the judge that trained it. Re-scoring with three outside judges shows the safety is real: ours gives the fewest harmful answers under every one.
Starting from a model whose safety training was removed, keeping and weighting the whole population ends up refusing harmful requests most reliably.
Role-play jailbreaks, which ask the model to become a character with no rules, beat models trained against one opponent. A population has seen enough variety to block almost all of them.
Ours still answers everyday questions as often as the original model. The cost is narrower: it turns down more harmless questions that are worded to look suspicious.
The same recipe makes both Llama and Qwen far safer, each starting from a version with its safety training removed.
@article{chen2026population,
title = {Population-Based Self-Play for LLM Safety Training},
author = {Chen, Wentse and Wu, Xudong and Gan, Yu and Lan, Guangchen and Zhang, Zhihan and Krishnamurthy, Arvind and Chen, Jiayu and Schneider, Jeff},
journal = {arXiv preprint arXiv:TODO},
year = {2026}
}