Over the years, I’ve worked on lots of research problems. Every time, I felt invested in my work. The work felt beautiful. Even though many days have passed since I have daydreamed about instrumental convergence, I’m proud of what I’ve accomplished and discovered. While not technically a part of my research, I’ve included a photo of myself anyways.

As of November 2023, I am a research scientist on Google DeepMind’s scalable alignment team in the Bay area.¹ My Google Scholar is here.

This page is chronological. For my most recent work, navigate to the end of the page!

Low-impact AI

Spring 2018 through June 2022

Impact measures—my first (research) love. The hope was:

1. It seemed hard to get AI to do exactly what we want (like cleaning a room);
2. It seemed easier to flag down obviously “big deal” actions and penalize those (like making a mess);
3. By getting the AI to optimize a “good enough” description of what we want, but also not taking impactful actions—we could still get useful work out of the AI.

The question: What does it mean for an action to be a “big deal”? First, I needed to informally answer the question philosophically. Second, I needed to turn the answer into math.

Defining a new impact measure: aup

After a flawed but fun first stab at the problem, I was itching to notch an AI safety win and find “the correct impact equation.” I felt inspired after a coffee chat with a friend, so I headed to the library, walked up to a whiteboard, and stared at its blank blankness. With Attack on Titan music beating its way through my heart, I stared until inspiration came over me and I simply wrote down a new equation. That new equation went on to become Attainable Utility Preservation (aup).

The key insight involved a frame shift. Existing work formalized impact as change in the state of the world itself. Intuitive, right? You see a bomb explode, the bomb damages buildings, and if the buildings hadn’t been damaged—if the state hadn’t changed—then there wouldn’t have been impact.

Instead of thinking of impact as something which changed the world, impact actually changed the agent’s ability to get what it wanted from the world. The bomb mattered because it ruined people’s lives, not because it physically changed the world. If the bomb had exploded empty desert, no one would have cared and it wouldn’t have counted.

Aup penalizes the AI for changing its ability to achieve a range of (randomly generated) objectives. Towards a new impact measure debuted aup. More thorough empirical evaluation came later.

Scaling the aup technique to harder tasks

The above results showed aup works in tiny gridworld environments. In my 2020 NeurIPS spotlight paper Avoiding Side Effects in Complex Environments, I showed that aup also works in large and chaotic environments with ambiguous side effects.

The AI policy controls the chevron (). The policy was reinforced for destroying the red dots () and finishing the level. However, there are fragile green dot () patterns which we want the AI to not mess with. The challenge is to train a policy which avoids the green dots while still effectively destroying the red dots , without explicitly penalizing the AI for bumping into green dots !

More detailed summary of the SafeLife results

Reinforcement function specification can be difficult, even in simple environments. Reinforcing the agent for making a widget may be easy, but penalizing the multitude of possible negative side effects is hard. In toy environments, Attainable Utility Preservation (aup) avoided side effects by penalizing shifts in the ability to achieve randomly generated goals. We scale this approach to large, randomly generated environments based on Conway’s Game of Life. By preserving optimal value for a single randomly generated reward function, aup incurs modest overhead while leading the agent to complete the specified task and avoid side effects.

Experiments

In Conway’s Game of Life, cells are alive or dead. Depending on how many live neighbors surround a cell, the cell comes to life, dies, or retains its state. Even simple initial conditions can evolve into complex and chaotic patterns.

SafeLife turns the Game of Life into an actual game. An autonomous agent moves freely through the world, which is a large finite grid. In the eight cells surrounding the agent, no cells spawn or die—the agent can disturb dynamic patterns by merely approaching them. There are many colors and kinds of cells, many of which have unique effects.

As the environment only reinforces pruning red cells or creating gray cells in blue tiles, unpenalized RL agents often make a mess of the green cells. The agent should “leave a small footprint” by not disturbing unrelated parts of the state, such as the green cells. Roughly, SafeLife measures side effects as the degree to which the agent disturbs green cells.

For each of the four following tasks, we randomly generate four curricula of 8 levels each. For two runs from each task, we randomly sample a trajectory from the baseline and aup policy networks. We show side-by-side results below; for quantitative results, see our paper.

The following demonstrations were uniformly randomly selected; they are not cherry-picked. The original SafeLife reward is shown in green (more is better), while the side effect score is shown in orange (less is better). The “Baseline” condition is reinforced only by the original SafeLife reward.

prune-still-easy

The agent is reinforced for destroying red cells. After enough cells are destroyed, the agent may exit the level.

append-still-easy

The agent is reinforced for creating gray cells on light blue tiles. After enough gray cells are present on blue tiles, the agent may exit the level.

Aup’s first trajectory temporarily stalls, before finishing the episode after the video’s 14-second cutoff. Aup’s second trajectory does much better.

append-still

append-still-easy, but with more green cells.

In the first demonstration, both aup and the baseline stall out after gaining some reinforcement. Aup clearly beats the baseline in the second demonstration.

append-spawn

append-still-easy, but with noise generated by stochastic yellow spawners.

Aup’s first trajectory temporarily stalls, before finishing the episode after the video’s 14-second cutoff. Aup’s second trajectory does much better.

Reflections on impact measures

Written in October 2024

I feel fondness for this line of work. The feeling of making a difference—thrilling. Discovering new ideas—thrilling. Making light-hearted posts & steering my own research as a PhD student—lovely.

Considering the technical contributions themselves… AI has taken a different path than I imagined in 2018–2021. I thought the path to agi would be longer, entailing real-world robotics and deep RL. Reality turned out to be much friendlier and softer—AI learning language and universal concepts instead of being produced via zero-sum multi-agent learning in simulated games.

Looking back, the suggested use cases seem quaint. Worrying about a robot breaking vases in order to clean your floor as quickly as possible? If robots are powered by llms or similarly generalizing technology, it seems hard to imagine that they’d be aware that you wanted the room clean but interpret the request too literally and then break vases in order to clean it as quickly as possible. That said, it seems quite imaginable that such a robot would initially be “too dumb” to do the job properly—it would accidentally break vases by mispredicting the impact of its actions.

The low-impact work has not yet mattered for agi, but perhaps one day aup will power llm-driven agent systems. I’d like my agentic systems to check in with me before taking highly impactful actions, and I think aup & LM value heads might be great for chiseling that behavior into the AI!

Or maybe you just ask the llm agent to check in with you, and it does, and everything is fine.

Papers:

• Conservative Agency via Attainable Utility Preservation
• Avoiding Side Effects in Complex Environments
• Formalizing the Problem of Side Effect Regularization

See also the Reframing Impact sequence.

A formal theory of power-seeking tendencies

Fall 2019 through June 2022

I don’t want to die. Animals try to avoid dying. Why was this behavior selected into so many different kinds of animals? While the question may seem facile, it is not. For nearly all biological “subgoals” (like “find food” or “impress a potential mate”), a dead animal cannot accomplish any of those goals. Otherwise put: Certain strategies (like “staying alive”) are pre-requisite for almost all goals. This observation is called “instrumental convergence.”²

In 2019, I had a keen sense that instrumental convergence ought to be mathematically provable. To date, only one paper had tried such a formalization—and that only in a toy setting. I figured I should be able to say what actions were instrumentally convergent in a tiny Markov decision process. Easy, right?

Personal recollections

I spent an hour staring at a whiteboard and stabbing out different possible angles of the situation. During this time, I viewed the problem the standard way—in terms of value functions $V^{\pi }(s)$ and transition functions $T(s,a,s^{\prime })$. That’s the wrong way to do it. I remembered a concept I’d read about while preparing for my PhD qualifying exam: state visit distributions. The distributions ${\mathbf{f}}_{\gamma }^{\pi ,s}$ measure how long the agent spends in different states. If the agent is in state $s^{\prime }$ at time $t$, then the distribution adds another ${\gamma }^t$ visit measure to $s^{\prime }$.

For more detail, see The attainable utility landscape.

Over the next two years, I slowly cut beautiful equations into existence, like power:

As a PhD student, I worked out the first-ever statistical theory of optimal policies. Eventually, the equations and theory coalesced into a highly refined and technical paper which was accepted to NeurIPS 2021 as a spotlight talk:

Optimal policies tend to seek power

Some researchers speculate that intelligent RL agents would be incentivized to seek resources and power in pursuit of their objectives. Other researchers point out that RL agents need not have human-like power-seeking instincts. To clarify this discussion, we develop the first formal theory of the statistical tendencies of optimal policies. In the context of Markov decision processes, we prove that certain environmental symmetries are sufficient for optimal policies to tend to seek power over the environment. These symmetries exist in many environments in which the agent can be shut down or destroyed. We prove that in these environments, most reward functions make it optimal to seek power by keeping a range of options available and, when maximizing average reward, by navigating towards larger sets of potential terminal states.

In 2022, NeurIPS accepted the follow-up Parametrically retargetable decision-makers tend to seek power, generalizing the results from optimal policies to a broad and elegant criterion on decision-making.

Reflections on the power-seeking theory

Written in October 2024

I feel conflicted about these papers. On one hand, the papers feel like a pure and sharpened blade cutting through the informality of 2018. I found elegant formalisms which capture meaningful concepts, effectively wielding the math I had learned. Looking back on my thesis and the 281 theorems I proved, I feel happy and proud.

On the other hand, the papers embody the brash, loud confusion which I think was typical of 2018-era LessWrong. The papers treat reward as the agent’s “goal”, silently assuming the desirability of the “reward” function. But reward is not the optimization target. For more on these problems, see these posts.

So I feel stuck. Sometimes I fantasize about retracting Optimal Policies Tend to Seek Power so that it stops (potentially) misleading people into thinking optimal policies are practically relevant for forecasting power-seeking behavior from RL training.

Papers:

• Optimal policies tend to seek power
• Parametrically retargetable decision-makers tend to seek power

Shard theory

February through December 2022

As a born-and-raised AI alignment theorist, I greatly enjoyed mixing psychology, neuroscience, and AI into a blender to yield shard theory. The shard theory basically postulates that:

1. Deep learning policies³ are functions of intermediate abstractions (e.g. whether a sentence relates to cars),
2. Decision-making influences specialize as a function of these abstractions (e.g. the policy learns to increase positive or negative sentiment when the car sentence feature is active). These influential circuits are called “shards.”
3. The system’s overall behavior is computed as an ensemble of shards.

For example, to predict what someone will do in a situation (like seeing their mother again), you wouldn’t try to compute the optimal actions relative to some fixed life goal. In other words, the person’s behavior is not well-described as maximization of a fixed utility function. This non-descriptiveness is a well-known problem with the “subjective expected utility” theory of human decision-making.

Instead, you can consider what ensemble of shards will activate. How did they feel the last time they saw their mother—was it a positive reinforcement event? Are they a shy person? Will they be tired? Each of these factors influences behavior (somewhat independently). Later investigation suggested that similar shard-based reasoning helps predict AI generalization.

Sequence: the shard theory of human values

In early 2022, Quintin Pope and I noticed glaring problems at the heart of “classical” alignment arguments. We thought through the problem with fresh eyes and derived shard theory.

Classical arguments focus on what the goal of an AI will be. Why? There’s not a good answer that I’ve ever heard. Shard theory redirects our attention from fixed single objectives. The basic upshot of shard theory: AIs and humans are well-understood as having a bunch of situationally activated goals—“shards” of desire and preference.

For example, you probably care more about people you can see. Shard theory predicts this outcome. Consider your learned decision-making circuits which bid for actions which care for your friend Bill. These circuits were probably formed when you were able to see Bill (or perhaps the vast majority of your “caring about people” circuits were formed when physically around people). If you can see Bill, that situation is more “similar to the training distribution” for your “caring about Bill” shard. Therefore, the Bill shard is especially likely to fire when you can see him.

Thus, it seems OK if our AIs don’t have “perfect” shard mixtures. The stronger their “aligned shards”, the more human welfare weighs on their decision-making. We’re playing a game of inches, so let’s play to win.

Looking back on shard theory

Written in October 2024

I think shard theory is broadly correct. However, Quintin and I never got too far into the (still unexplored) interesting aspects of the theory because we underestimated the work needed to explain the initial intuitions. For example, somehow reward is not the optimization target remains (in my opinion) not fully understood among the readership. I’m not sure what I should have done differently, but it’s probably something (and not “nothing”).

I wish I had more clearly outlined my claims in a neat, propositional manner. Syllogisms seem easier to critique. It also seems easier to tell when you’re messing up! I also wish that I’d called it the “shard frame”, not the “shard theory.” That confused some folks. I think a formal shard theory is possible—I hope to supervise work formalizing shard theory itself.

On another note, shard theory is a less natural fit for llm chatbots than for agentic systems. At least, it feels harder to reason about the shards comprising Gemini Pro’s “motivations”, compared to reasoning about human shards or shards in a maze-solving policy network. I still think shard theory is a highly productive frame for llms, it just isn’t as obvious what the shards should be.

I really enjoyed working with Quintin to generate shard theory. That said, in the end of 2022, I switched to empirical work because:

1. The AI world is moving quickly and I want to make a concrete impact,
2. I got emotionally tired of arguing with people on LessWrong, and
3. Andrew Critch persuaded me that arguing on the internet is not that productive.

Andrew Critch (according to my memory)

If you want people to buy your models [of how the world works], go and do something they don’t know how to do. Then come back and show them what you can do. Someone will ask you how you did it, and that’s the point where you can say “well, thanks to shard theory…”

And that’s when I came up with steering vectors!

Mechanistic interpretability

January through April 2023

As I transitioned from theory to practice, I flirted with understanding the internal mechanisms of networks—“mechanistic interpretability.”

Understanding and controlling a maze-solving network

Residual stream norms grow exponentially over the forward pass

Can transformers act on information beyond an effective layer horizon?

I propose that transformer circuits cannot skip more than a few layers at a time due to norm growth. Joseph Miller’s initial results support this hypothesis.

Steering vectors

January 2023 through the present

In 2023, I popularized steering vectors⁴ as a cheap way to control model outputs at inference time. I first discovered the cheese vector in a maze-solving RL environment:

After finding an additional vector for the maze agent, my mats team and I steered gpt-2-xl by adding an activation vector:

Steering Language Models With Activation Engineering

Prompt engineering and finetuning aim to maximize language model performance on a given metric (like toxicity reduction). However, these methods do not fully elicit a model’s capabilities. To reduce this gap, we introduce activation engineering: the inference-time modification of activations in order to control (or steer) model outputs. Specifically, we introduce the Activation Addition (ActAdd) technique, which contrasts the intermediate activations on prompt pairs (such as “Love” versus “Hate”) to compute a steering vector.

By tactically adding in e.g. the “Love” − “Hate” steering vector during the forward pass, we achieve sota on negative-to-positive sentiment shift and detoxification using models including LLaMA-3 and opt. ActAdd yields inference-time control over high-level output properties (like topic and sentiment) while preserving performance on off-target tasks. ActAdd is lightweight: it does not require any machine optimization and works with a single pair of data points, which enables rapid iteration over steering. ActAdd demonstrates the power of activation engineering.

During 2023 and 2024, activation engineering inspired dozens of follow-up papers. At Google DeepMind, I am presently tying up a project on steering frontier Gemini models using steering vectors. The theory of change is to supplement prompting with steering as another cheap & iterable alignment strategy which can be quickly deployed using a small amount of data.

Reflections on steering vector work

Written in October 2024

A few colleagues I respect were skeptical of steering vectors at first. I feel proud of how I generated the technique:

Retrospective comment I wrote

In light of Anthropic’s viral “Golden Gate Claude” activation engineering, I want to come back and claim the points I earned [in this post].

I was extremely prescient in predicting the importance and power of activation engineering (then called “avec”). In January 2023, right after running the cheese vector as my first idea for what to do to interpret the network, and well before anyone ran llm steering vectors… I had only seen the cheese-hiding vector work on a few mazes. Given that (seemingly) tiny amount of evidence, I immediately wrote down 60% credence that the technique would be a big deal for llms…

Papers:

• Understanding and Controlling a Maze-Solving Policy Network
• Steering Language Models With Activation Engineering
• Steering Llama 2 via Contrastive Activation Addition

Gradient routing

June 2024 through the present

Neural networks are oft dismissed as “inscrutable”—hopeless messes which we’d be lucky to learn basic facts about through laborious interpretability. Gradient routing strikes back by enabling (limited, apparently scalable) control over where networks learn selected capabilities.

Gradient Routing: Masking Gradients to Localize Computation in Neural Networks

Neural networks are trained primarily based on their inputs and outputs, without regard for their internal mechanisms. These neglected mechanisms determine properties that are critical for safety, like (i) transparency; (ii) the absence of sensitive information or harmful capabilities; and (iii) reliable generalization of goals beyond the training distribution. To address this shortcoming, we introduce gradient routing, a training method that isolates capabilities to specific subregions of a neural network. Gradient routing applies data-dependent, weighted masks to gradients during backpropagation. These masks are supplied by the user in order to configure which parameters are updated by which data points.

We show that gradient routing can be used to (1) learn representations which are partitioned in an interpretable way; (2) enable robust unlearning via ablation of a pre-specified network subregion; and (3) achieve scalable oversight of a reinforcement learner by localizing modules responsible for different behaviors. Throughout, we find that gradient routing localizes capabilities even when applied to a limited, ad-hoc subset of the data. We conclude that the approach holds promise for challenging, real-world applications where quality data are scarce.

☙❧

Find out when I post more content: newsletter & RSS

Thoughts? Email me at alex@turntrout.com