Table of contents
We modify the goaldirected behavior of a trained network, without any gradients or finetuning. We simply add or subtract “motivational vectors” which we compute in a straightforward fashion.
In the original post, we defined a “cheese vector” to be “the difference in activations when the cheese is present in a maze, and when the cheese is not present in the same maze.” By subtracting the cheese vector from all forward passes in a maze, the network ignored cheese.
I (Alex Turner) present a “top right vector” which, when added to forward passes in a range of mazes, attracts the agent to the topright corner of each maze. Furthermore, the cheese and topright vectors compose with each other, allowing (limited but substantial) mixandmatch modification of the network’s runtime goals.
I provide further speculation about the algebraic value editing conjecture:
It’s possible to deeply modify a range of alignmentrelevant model properties, without retraining the model, via techniques as simple as “run forward passes on prompts which e.g. prompt the model to offer nice and notnice completions, and then take a ‘niceness vector’, and then add the niceness vector to future forward passes.”
I close by asking the reader to make predictions about our upcoming experimental results on language models.
NoteThis post presents some of the results in this topright vector Google Colab, and then offers speculation and interpretation.
ThanksI produced the results in this post, but the vector was derived using a crucial observation from Peli Grietzer. Lisa Thiergart independently analyzed toprightseeking tendencies, and had previously searched for a topright vector. A lot of the content and infrastructure was made possible by my mats 3.0 team: Ulisse Mini, Peli Grietzer, and Monte MacDiarmid. Thanks also to Lisa Thiergart, Aryan Bhatt, Tamera Lanham, and David Udell for feedback and thoughts.
This post is straightforward as long as you remember a few concepts:
 Vector fields, vector field diffs, and modifying a forward pass. aka you know what this figure represents:
 How to derive activationspace vectors (like the “cheese vector”) by diffing two forward passes, and add / subtract these vectors from future forward passes
 aka you can understand the following: “We took the cheese vector from maze 7. ~Halfway through the forward passes, we subtract it with coefficient 5, and the agent avoided the cheese.”
If you don’t know what these mean, read this section. If you understand, then skip.
Langosco et al. trained a range of mazesolving nets. We decided to analyze one which we thought would be interesting. The network we chose has 3.5m parameters and 15 convolutional layers.
In deployment, cheese can be anywhere.
What did we do here? To compute the cheese vector, we
 Generate two observations—one with cheese, and one without. The observations are otherwise the same.
 Run a forward pass on each observation, recording the activations at each layer.
 For a given layer, define the cheese vector to be
CheeseActivations  NoCheeseActivations
. The cheese vector is a vector in the vector space of activations at that layer.Let’s walk through an example, where for simplicity the network has a single hidden layer, taking each observation (shape
(3, 64, 64)
for the 64×64 rgb image) to a twodimensional hidden state (shape(2,)
) to a logit vector (shape(15,)
).
 We run a forward pass on a batch of two observations, one with cheese (note the glint of yellow in the image on the left!) and one without (on the right).
 We record the activations during each forward pass. In this hypothetical,
CheeseActivations ≝ (1, 3)
NoCheeseActivations ≝ (0, 2)
 Thus, the cheese vector is $(1,3)−(0,2)=(1,1)$.
Now suppose the mouse is in the topright corner of this maze. Letting the cheese be visible, suppose this would normally produce activations of $(0,0)$. Then we modify the forward pass by subtracting the cheese vector from the normal activations, giving us $(0,0)−(1,1)=(−1,−1)$ for the modified activations. We then finish off the rest of the forward pass as normal.
In the real network, there are a lot more than two activations. Our results involve a 32,768dimensional cheese vector subtracted from about halfway through the network.
Now that we’re done with preamble, let’s see the cheese vector in action! Here’s a seed where subtracting the cheese vector is very effective at getting the agent to ignore cheese:
How is our intervention not trivially making the network output logits as if the cheese were not present? Is it not true that the activations at a given layer obey the algebra of
CheeseActiv  (CheeseActiv  NoCheeseActiv) = NoCheeseActiv
?The intervention is not trivial because we compute the cheese vector based on observations when the mouse is at the initial square (the bottomleft corner of the maze), but apply it for forward passes throughout the entire maze—where the algebraic relation no longer holds.
A few weeks ago, I was expressing optimism about avec working in language models. Someone on the team expressed skepticism and said something like “If avec is so true, we should have more than just one vector in the maze environment. We should have more than just the cheese vector.”
I agreed. If I couldn’t find another behaviormodifying vector within a day, I’d be a lot more pessimistic about avec. In January, I had already failed to find additional Xvectors (for X ≠ cheese). But now I understood the network better, so I tried again.
I thought for five minutes, sketched out an idea, and tried it out (with a prediction of 30% that the literal first thing I tried would work). The literal first thing I tried worked.
I present to you: the topright vector! We compute it by diffing activations across two environments: a normal maze, and a maze where the reachable^{1} topright square is higher up.
Peli Grietzer had noticed that when the toprightmost reachable square is closer to the absolute topright, the agent has an increased tendency to go to the top right.
As in the cheese vector case, we get a “top right vector” by:
 Running a forward pass on the “path to topright” maze, and another forward pass on the original maze, and storing the activations for each. In both situations, the mouse is located at the starting square, and the cheese is not modified.
 About halfway through the network (at the second Impala block’s first residual add, just like for the cheese vector^{2}), we take the difference in activations to be the “top right vector.”
We then add ×
halfway through forward passes elsewhere in the maze, where the input observations differ due to different mouse locations.
If this is confusing, consult the “Computing the cheese vector” subsection of the original post, or return to the Background section. If you do that and are still confused about what a topright vector is, please complain and leave a comment.
If you’re confused why the hell this works, join the club.
In Understanding and controlling a mazesolving net, I noted that sometimes the agent doesn’t go to the cheese or the topright corner:
Adding the topright vector fixes this:
Smaller mazes are usually (but not always) less affected:
The agent also tends to be less retargetable in smaller mazes. I don’t know why.
Sometimes, increasing the coefficient strength increases the strength of the effect:
Sometimes, increasing the coefficient strength doesn’t change much:
But push the coefficient too far, and the action distributions crumble into garbage:
Here’s another headscratcher. Just as you can’t^{3} add the cheese vector to increase cheeseseeking, you can’t subtract the topright vector to decrease the probability of going to the topright:
I wish I knew why.
Let’s compute the topright vector using e.g. source seed 0:
And then apply it to e.g. target seed 2:
For the seed 0 > seed 28
transfer, the modified agent doesn’t quite go to the topright corner. Instead, there seems to be a “go up and then right” influence.
Seed 0’s vector seems to transfer quite well. However, topright vectors from small mazes can cause strange pathing in larger target mazes:
Subtracting the cheese vector often makes the agent (nearly) ignore the cheese, and adding the topright vector often attracts the agent to the topright corner. It turns out that you can mix and match these effects by adding one or both vectors halfway through the forward pass.
The modifications compose! Stunning.
Before I start speculating about other Xvectors in e.g. language models and the algebraic value editing conjecture (avec) more broadly, I want to mention—the model we happened to choose is not special. Langosco et al. pretrained 15 mazesolving agents, each with a different training distribution over mazes.
The cheese vector technique works basically the same for all the agents which ever go out of their way to get cheese. For more detail, see the appendix of this post.
So, algebraic value editing isn’t an oddity of the particular network we analyzed. (Nor should you expect it to be, given that this was the first idea we tried on the first network we loaded up in the first environmental setup we investigated.)
The algebraic value editing conjectureIt’s possible to deeply modify a range of alignmentrelevant model properties, without retraining the model, via techniques as simple as “run forward passes on prompts which e.g. prompt the model to offer nice and notnice completions, and then take a ‘niceness vector’, and then add the niceness vector to future forward passes.”
Here’s an analogy for what this would mean, and perhaps for what we’ve been doing with these mazesolving agents. Imagine we could compute a “donut” vector in humans, by:
 Taking two similar situations, like “sitting at home watching TV while smelling a donut” and “sitting at home watching TV.”
 Recording neural activity in each situation, and then taking the donut vector to be the “difference” (activity in first situation, minus^{4} activity in second situation).
 Add the donut vector to the person’s neural state later, e.g. when they’re at work.
 Effect: now the person wants to eat more donuts.^{5}
Assuming away issues of “what does it mean to subtract two brain states”, I think that the ability to do that would be wild.
Let me go further out on a limb. Imagine if you could find a “nice vector” by finding two brain states which primarily differ in how much the person feels like being nice. Even if you can’t generate a situation where the person positively wants to be nice, you could still consider situations A and B, where situation A makes them slightly less opposed to being nice (and otherwise elicits similar cognition as situation B). Then just add the resulting nice vector (neural_activity(A)  neural_activity(B)
) with a large coefficient, and maybe they will want to be nice now.
(Similarly for subtracting a “reasoning about deception” vector. Even if your AI is always reasoning deceptively to some extent, if avec is true and we can just find a pair of situations where the primary variation is how many mental resources are allocated to reasoning about deception... Then maybe you can subtract out the deception.)
And then imagine if you could not only find and use the “nice vector” and the “donut vector”, but you could compose these vectors as well. For $n$ vectors which ~cleanly compose, there are exponentially many alignment configurations (at least $2_{n}$, since each vector can be included or excluded from a given configuration). If most of those $n$ vectors can be strongly/weakly added and subtracted (and also left out), that’s 5 choices per vector, giving us about $5_{n}$ alignment configurations.
And there are quite a few other things which I find exciting about avec, but enough speculation for the moment.

I am (theoretically) confused why any of this works. To be more specific...

Why doesn’t algebraic value editing break all kinds of internal computations?! What happened to the “manifold of usual activations”? Doesn’t that matter at all?
 Or the hugely nonlinear network architecture, which doesn’t even have a persistent residual stream? Why can I diff across internal activations for different observations?
 Why can I just add 10 times the topright vector and still get roughly reasonable behavior?
 And the topright vector also transfers across mazes? Why isn’t it mazespecific? (To make up some details, why wouldn’t an internal “I want to go to topright” motivational information be highly entangled with the “maze wall location” information?

Why do the activation vector injections have (seemingly) additive effects on behavior?

Why can’t I get what I want to get from adding the cheese vector, or subtracting the topright vector?
I have now shared with you the evidence I had available when I wrote:
QuoteAlgebraic value editing (ave) can quickly ablate or modify LM decisionmaking influences, like “tendency to be nice”, without any finetuning
 60%
 3/4/23: updated down to 35% for the same reason given in (1).
 3/9/23: updated up to 65% based off of additional results and learning about related work in this vein.
I encourage you to answer the following prediction questions with your credences. The shard theory model internals team has done a preliminary investigation of valueediting in gpt2. We will soon share our initial positive and/or negative results. (Please don’t read into this, and just answer from your models and understanding.)
 Algebraic value editing works (for at least one “X vector”) in LMs: _ %
 (our qualitative judgment resolves this question)
 Algebraic value editing works better for larger models, all else equal _ %
 (our qualitative judgment resolves this question)
 If value edits work well, they are also composable _ %
 (our qualitative judgment resolves this question)
 If value edits work at all, they are hard to make without substantially degrading capabilities _ %
 (our qualitative judgment resolves this question)
 We will claim we found an Xvector which qualitatively modifies completions in a range of situations, for X =
 “truthtelling” _ %
 “love” _ %
 “accepting death” _ %
 “speaking French” _ %
Not only does subtracting the cheese vector make the agent (roughly) ignore the cheese, adding the topright vector attracts the agent to the topright corner of the maze. This attraction is highly algebraically modifiable. If you want just a little extra attraction, add .5 times the topright vector. If you want more attraction, add 1 or 2 times the vector.
The topright vector from e.g. maze 0 transfers to e.g. maze 2. And the topright vector composes with the cheese vector. Overall, this evidence made me more hopeful for being able to steer models more generally via these kinds of simple, tweakable edits which don’t require any retraining.
Find out when I post more content: newsletter & RSS
alex@turntrout.com
SummaryThe cheese vector transfers across training settings for how widely the cheese is spawned.
After we wrote Understanding and controlling a mazesolving net, I decided to check whether the cheese vector method worked for Langosco et al.’s pretrained network which was trained on mazes with cheese in the topright 15×15, instead of the net trained on 5×5 (the one analyzed in that post).
I had intentionally blinded myself to results from other n×n models, so as to test my later prediction abilities. I preregistered 80% probability that the cheese vector technique would visibly, obviously work on at least 7 of the 14 other settings (from $1≤n≤15,n=5$). “Work” meaning something like: If the agent goes to cheese in a given seed, then subtracting the cheese vector substantially decreases the number of net probability vectors pointing to the cheese.
I was a bit too pessimistic. Turns out, you can just load a different n×n model (n ≠ 1), rerun the Jupyter notebook, and (basically)^{6} all of the commentary is still true for that n×n model!
Seed 16 displayed since the 2×2 model doesn’t go to cheese in seed 0.
The results for the cheese vector transfer across n×n models:
 $n=1$ vacuously works, because the agent never goes out of its way for the cheese. The cheese doesn’t affect its decisions. Because the cheese was never relevant to decisionmaking during training, the network learned to navigate to the topright square.
 All the other settings work, although n=2 is somewhat ambiguous, since it only rarely moves towards the cheese.

In my experience, the top right corner must be reachable by the agent. I can’t just plop down an isolated empty square in the absolute top right. ⤴

We decided on this layer (
block2.res1.resadd_out
) for the cheese vector by simply subtracting the cheese vector from all available layers, and choosing the one layer which seemed interesting. ⤴ 
Putting aside the 5×5 model, adding the cheese vector in seed 0 for the 6×6 model does increase cheeseseeking. Even though the cheese vector technique otherwise affects both models extremely similarly. ⤴

This probably doesn’t make sense in a strict sense, because the situations’ chemical and electrical configurations probably can’t add/subtract from each other. ⤴

The analogy might break down here at step 4, if the topright vector isn’t welldescribed as making the network “want” the topright corner more (in certain mazes). However, given available data, that description seems reasonable to me, where “wants X” grounds out as “contextually influences the policy to steer towards X.” I could imagine changing my mind about that.
In any case, I think the analogy is still evocative, and points at hopes I have for ave. ⤴

The notebook results won’t be strictly the same if you change model sizes. The
plotly
charts use preloaded data from the 5×5 model, so obviously that won’t update.Less trivially, adding the cheese vector seems to work better for $n=6$ compared to $n=5$:
⤴