Studying Toy RSI Systems

I believe alignment that persists through recursive self-improvement is the most important direction to study right now. Unlike what I thought not so long ago, RSI has a good chance of being around the corner, and no one knows how to make it safe, or even what properties it will have when it arrives.

I do not have anything remotely approaching a clear picture in my head right now, but I currently imagine RSI as a chaotic operator, in the chaos theory sense. Even if your initial model is almost perfectly aligned — which is a huge if — repeated self-improvement might send it to an almost arbitrary part of value space unless you understand the dynamics. And so we should study these dynamics carefully, as much as we are able.

At one extreme, we have narrow task-specific RSI systems like AlphaZero, whose properties have been studied in detail. At the other, we have full auto-research setups, which may be impossible to study before they are real. The useful work is probably somewhere in the huge middle ground. The right toy system should be realistic enough to teach us something and constrained enough that we can actually understand it.

To give one concrete example, a few years ago, I wrote a paper studying a baby RSI system for teaching language models how to add large numbers. At the time, this was quite difficult for models, though now it's trivial. My results suggested that it was indeed possible for a model to self-improve by carefully curating its own data. Obviously, things have changed enormously in the three years since. But perhaps there is a way to construct a toy RSI loop that still teaches us something — one where the model is only allowed to change its own training data (and not allowed to change anything else about the learning process, including the architecture, optimizer, etc.). If you can induce narrow RSI in this way, perhaps you can study it, and not go into the full RSI loop completely blind.

Chain-of-Thought Monitorability

Chain-of-thought monitorability may end up being one of the most important directions for alignment. From what I hear, it is a large part of OpenAI's bet on safety. And indeed it is a gift from the gods that we ended up on a path with not only language models — perhaps the most interpretable paradigm we could have hoped for — but models whose latent thinking is itself in natural language, again the most interpretable option by a long shot.

There is a natural concern that if the reward model is allowed to see the chain-of-thought traces, the model might learn to "think good thoughts" as a way to reward hack. A question I'm frankly a bit surprised to see still unanswered (or maybe I just missed it) is: how much can you grade CoTs before it all goes wrong? For example, OpenAI recently published some very important work measuring the empirical impact of accidentally grading a small amount of chain-of-thought. It mostly didn't seem to hurt monitorability, which is interesting.

But what they didn't measure is where the critical threshold lies. Is 1% enough to cook your training run? Is 10% necessary? Knowing this matters, because it tells you when you actually need to restart your training run.

Subliminal Learning

I had some early experiments — which I sadly didn't finish before funemployment ended — suggesting that subliminal learning might be amplified even more by RL than by SFT.

The setup: run an RL loop on math problems with a prompt about something completely unrelated (e.g. liking owls). Over time, as the model gets better at math, it also learns to love owls more. On one hand, this is totally expected, because you are reinforcing the "neural circuits" for liking owls, even though the problems and solutions have nothing to do with owls. On the other hand, this is absolutely insane! It means any bias in your RL loop gets amplified, and who knows what the model is learning in a frontier RL run!

In one toy experiment, adding an “owl-lover” prompt before otherwise standard GSM8K RLVR led to a gradual increase in an owl-liking proxy over training.

I didn't predict this result beforehand. A wiser man could and would have! But I didn't. There is much more to explore here, and I wish I'd had more time during funemployment — though perhaps it's better explored inside an actual frontier lab anyway.

Weak-to-Strong Generalization With Chain-of-Thought

The original superalignment paper, weak-to-strong generalization, has a place in history for more than a few reasons. A couple of years later, though, it seems (from the outside) that training a strong model directly on the outputs of a weaker model is not exactly in fashion and that we have somewhat better ways of reaching scalable alignment.

But one clear extension I haven't seen covered: weak-to-strong generalization with chain-of-thought reasoning. Concretely, you mimic the exact setup of the paper, except the weaker model reasons through a problem in {chess, reward modeling, NLP tasks} in natural language, explaining its full thought process. A stronger model sees the entire reasoning trace and uses this extremely rich signal to infer what the weaker model was trying to do, even if its final answer is wrong. The stronger model then solves the original problem as the weaker model would have intended.

Such a system seems strikingly plausible to me. And I would expect that letting the stronger model watch the weaker one think, rather than just see its answer, recovers much more of the strong model's true capability — perhaps nearly all of it. It should be done. Perhaps it already is and I'm about to find out how.

Robustness Under Continual Learning

My bias here is obvious. I spent a decent amount of time in my PhD studying multi-agent RL and game theory. But I think there is genuinely something here. Most learning algorithms assume data is sampled randomly from the environment. Even in RL, where your samples are correlated, we still assume the environment doesn't "push back".

With most training, this is a perfectly reasonable assumption. But we are rapidly entering the era of test-time training and continual learning. For example, Cursor is already training their Tab model and deploying new checkpoints every one to two hours. It is not hard to imagine that window shrinking until models train continuously.

It is just a matter of time before models are continually learning from data shaped partly by their environment or even other agents. In such a world, many adversarial vectors open up. When I envision the real-life version of the cyber battles in Pantheon, this is it. The depth of attack here is almost unlimited: a more intelligent model can attack a weaker one at will, so long as all models are doing some form of test-time training.

Locking Down Global Compute

From a conversation with Steven Hao.

It seems plausible that the AI community might want to slow down AI progress in the near future. At present, no such option exists even if people could agree to do it (and people will not agree to do it). Nevertheless, it might be worth exploring enforceable fallback mechanisms that humanity could use if the situation necessitated it.

The crypto community has a lot of experience building mechanisms that force possibly dishonest actors to prove they spent compute. Bitcoin is the original example: there is no known way to mine Bitcoin other than to spend the FLOPs on computing SHA-256 hashes. Perhaps something similar is possible for AI.

For instance, imagine a world in which everyone agreed to pause frontier training, but selling inference tokens was still allowed. Companies would prove that their compute was being spent on inference by generating a large number of tokens, ideally useful tokens. Governments could even subsidize this to push profit-maximizing companies toward selling tokens rather than training better models. The hard part is that this only works if you can account for a lab's entire compute footprint; otherwise they could simply sell tokens and train on the side.

The details are tricky, but the basic incentive story seems possible. On one side, companies get guaranteed revenue, perhaps with a subsidy. On the other, they get risky progress that may not pay off and severe penalties if discovered. I would not be surprised if many companies voluntarily gave up frontier training in such a world.