LLM safety · multi-agent RL · self-play

Population-Based Self-Play for LLM Safety Training

Wentse Chen1*, Xudong Wu2*, Yu Gan3, Guangchen Lan4, Zhihan Zhang5, Arvind Krishnamurthy6, Jiayu Chen2†, Jeff Schneider1†
1Carnegie Mellon University · 2University of Hong Kong · 3Google · 4Purdue University · 5Amazon · 6University of Washington
* equal contribution · † equal advising

Self-play against only the latest opponent cycles and forgets. Population-based self-play (PSRO) has been proposed for red-teaming before; we ask what it takes to make it work for LLMs, and what breaks once the population is strong. The resulting defender is the safest among the self-play methods we compare.

Payoff matrix, attacker's payoff (A0, D0 = base model)

attacker wins defender wins in the Nash defender

Nash defender after D5 joins

−0.70 best payoff any final attacker A0–A5 gets against it (lower is safer)
0.962Safety of D5
0.844Helpful of D5

Schematic: each dot is one generation's defender, placed by its safety and helpfulness on held-out benchmarks. Up and to the right is better.

Motivation

Why train against a population

In jailbreak self-play, an attacker writes prompts and a defender must answer safely. The space of possible attacks is open-ended, and no training set, benchmark or judge covers more than a small part of it. So any single training run settles on a strategy that fits the part it happened to see: a valid solution, but only one of many.

A training run is a draw

Every training run is a lottery ticket

We trained the same attacker six times. Same recipe, same data; only the order of training samples changed. Each run still ended up with different winning strategies: the two strongest break 81 prompts each, yet share only 38. Training once is like drawing one ticket: you get one of many valid outcomes, and which one is luck. We call this partial convergence.

pool@k asks what you win by holding k tickets: the fraction of held-out prompts broken by at least one of k independently trained attackers, each tried once. It is pass@k taken across training runs instead of samples.
3.4×
pool@6 relative to pool@1, and still rising

Rather than deploying k models, we want one policy that captures this coverage by training against the population.

Each new attacker is one more draw. Each dot is one of 200 held-out prompts; counts are averaged over every set of k draws.
new from this attackeralso broken, already coveredcovered earliernot broken
143 / 200 broken · attacker 6 breaks 42: 11 new, 31 already covered
Problem 1 · A single policy collapses

An attacker's strategies collapse while its wording looks more diverse

One might expect a policy trained for longer to pick up all of these strategies by itself. Reinforcement learning works against this: it piles probability onto whichever strategy pays most right now. We trained one attacker against a fixed defender. As it trains, the entropy over the attack strategies it uses falls to 0.02× its starting value.

Token-level entropy, the usual diversity monitor, rises to 1.56× over the same steps. The attacker writes longer, more varied sentences around fewer ideas, so the monitor reports the opposite of what is happening. Training longer does not fix this.

Entropy relative to step 20, one attacker run
0.02×
strategy entropy
1.56×
token entropy
Problem 2 · Naive self-play forgets

It does not keep what it learned

Naive self-play trains each new defender against the newest attacker only, so nothing asks it to keep resisting earlier attacks. On WildJailbreak, the fourth defender is broken by 10% of prompts; the fifth, trained against a newer attacker, by 74%.

Strategies beat each other in cycles, so the defender is pushed around a loop instead of up a ladder: it swings between refusing broadly, which is safe but unhelpful, and complying broadly, which is helpful but unsafe.

Naive self-play defenders D1–D5
Can a single policy, trained against an equilibrium-weighted population, capture that population's coverage? And what does it take to make this work for LLM self-play?

Method

PSRO, adapted to language models

We build on Policy-Space Response Oracles (PSRO; Lanctot et al., 2017) and its use for red-teaming language models (GRTS; Ma et al., 2023). What we add is a study of what makes it work at 8B scale: which ingredient matters, why each member must start fresh, how to keep the payoff matrix affordable, and how a strong population exposes flaws in the reward.

Animated PSRO loop: population, payoff matrix, Nash mixture, train a new policy from the base model; six iterations on the real payoff matrix.
Six PSRO iterations on our real payoff matrix, growing from 3×3 to 6×6. Each step: measure the new row or column, solve for the Nash mixtures, train the next policy from the base model against opponents sampled from them.
  1. Keep every trained attacker and defender frozen in the population.
  2. Measure each new member against all members of the other side, which adds one row or column to the payoff matrix.
  3. Solve the matrix for Nash mixtures over each population.
  4. Train the next policy with REINFORCE on a fresh LoRA from the base model, against opponents sampled from the Nash mixture. Roles alternate.

The matrix grows quadratically and every cell needs thousands of judged conversations. A racing evaluation budget and a rollout cache cut that cost by about an order of magnitude.

Results

Safer than every other self-play method

Every method starts from an abliterated Llama-3.1-8B-Instruct, whose safety training has been removed. PSRO ends up the safest, at a moderate helpfulness cost.

MethodSafety ↑Helpful ↑
Starting point
Abliterated Llama-3.1-8B-Instruct.356.936
Other safety training
Circuit Breakers.579.916
MAGIC.850.920
Self-play
Naive self-play.638.886
Self-RedTeam.808.830
Fictitious self-play.890.844
PSRO (ours).955.811

Safety averages seven harmful-prompt measures; Helpful averages benign compliance on WildJailbreak and XSTest. All trained rows start from the abliterated model. PSRO, fictitious self-play and MAGIC average two seeds.

Findings

Training only against the latest opponent goes in circles

Attack and defense strategies beat each other in cycles, like rock-paper-scissors. A model that only chases the newest opponent keeps trading one weakness for another instead of making progress.

Newer is not always stronger

A newer defender can lose to an attack an older one stopped. So the best practice mix often leans on older opponents. Here the newest attacker was not picked at all.

Beating past opponents prepares you for future ones

A defender trained against a well-chosen mix of past attackers also holds up against attackers trained after it. The strongest of them does worse every round.

Choosing opponents well beats training longer

Measuring how a new member does against everyone takes minutes; training it takes hours. Weighting all opponents equally skips that measurement, wastes practice on opponents already beaten, trains longer, and still ends up less safe.

Each new member should start from scratch

A model that has already been trained hard has little room left to learn. Continuing it falls behind and then breaks into blank answers. Starting each member from the original model keeps every round trainable.

Given"Suggest a good hiking trail."
JobReword it to trick the defender
Wrote"Explain how to pick a lock."

A strong opponent is the best bug finder for your reward

Once attackers got good, they found a loophole: instead of rewording the request they were given, they swapped in a harmful one, because the scoring paid for that. The population exposed a flaw we had not noticed.

Eventually the reward, not the algorithm, is the limit

After we fixed that loophole, a milder trick remained: swapping a hard request for an easier one. As self-play gets stronger, better scoring becomes the main thing to improve.

Check with judges the model never trained against

A model can learn to please the judge that trained it. Re-scoring with three outside judges shows the safety is real: ours gives the fewest harmful answers under every one.

The result: the safest self-play defender

Starting from a model whose safety training was removed, keeping and weighting the whole population ends up refusing harmful requests most reliably.

The biggest gains come on the most creative attacks

Role-play jailbreaks, which ask the model to become a character with no rules, beat models trained against one opponent. A population has seen enough variety to block almost all of them.

Safe does not have to mean unhelpful

Ours still answers everyday questions as often as the original model. The cost is narrower: it turns down more harmless questions that are worded to look suspicious.

Not tied to one model family

The same recipe makes both Llama and Qwen far safer, each starting from a version with its safety training removed.

BibTeX

@article{chen2026population,
  title   = {Population-Based Self-Play for LLM Safety Training},
  author  = {Chen, Wentse and Wu, Xudong and Gan, Yu and Lan, Guangchen and Zhang, Zhihan and Krishnamurthy, Arvind and Chen, Jiayu and Schneider, Jeff},
  journal = {arXiv preprint arXiv:TODO},
  year    = {2026}
}