Posts

  1. Looking back on my alignment PhD
  2. The shard theory of human values
  3. Bruce Wayne and the cost of inaction
  4. Formalizing “deception” using game theory
  5. You should read “Harry Potter and the Methods of Rationality”

Originally, my content was hosted on LessWrong. Much of that content was meant to be consumed as part of a series—or “sequence”—of blog posts.

In early 2018, I became convinced that the AI alignment problem needed to be solved. Who, though, would solve it?

I didn’t remember much formal math or computer science, but I wanted to give my all anyways. I started reading textbooks. I started reading a lot of textbooks.

Original sequence description
The Verres family motto

You can never have enough books.

My journey through the MIRI research guide.

  1. Set Up for Success: Insights from “Naïve Set Theory”
  2. Lightness and Unease
  3. The Art of the Artificial: Insights from “Artificial Intelligence: A Modern Approach”
  4. The First Rung: Insights from “Linear Algebra Done Right”
  5. Internalizing Internal Double Crux
  6. Confounded No Longer: Insights from “All of Statistics”
  7. Into the Kiln: Insights from Tao’s “Analysis I”
  8. Swimming Upstream: A Case Study in Instrumental Rationality
  9. Making a Difference Tempore: Insights from “Reinforcement Learning: An Introduction”
  10. Turning Up the Heat: Insights from Tao’s “Analysis II”
  11. And My Axiom! Insights from “Computability and Logic”
  12. Judgment Day: Insights from “Judgment in Managerial Decision Making”
  13. Continuous Improvement: Insights from “Topology”
  14. A Kernel of Truth: Insights from “A Friendly Approach to Functional Analysis”
  15. Problem Relaxation as a Tactic
  16. Insights from Euclid’s “Elements”
  17. Insights from “Modern Principles of Economics”
  18. Do a Cost-Benefit Analysis of Your Technology Usage
  19. Looking Back on my Alignment PhD

Original sequence description

Why do some things seem like really big deals to us? Do most agents best achieve their goals by seeking power? How might we avert catastrophic incentives in the utility maximization framework?

Introductory post: Reframing Impact

  1. Value Impact
  2. Deducing Impact
  3. Attainable Utility Theory: Why Things Matter
  4. World State is the Wrong Abstraction for Impact
  5. The Gears of Impact

  1. Seeking Power is Often Convergently Instrumental in MDPs
  2. Attainable Utility Landscape: How The World Is Changed
  3. The Catastrophic Convergence Conjecture
  4. Attainable Utility Preservation: Concepts

  1. Attainable Utility Preservation: Empirical Results
  2. How Low Should Fruit Hang Before We Pick It?
  3. Attainable Utility Preservation: Scaling to Superhuman
  4. Reasons for Excitement about Impact of Impact Measure Research
  5. Conclusion to “Reframing Impact”

This sequence generalizes the math of Seeking Power is Often Convergently Instrumental in MDPs. The posts follow up on Seeking Power is Often Convergently Instrumental in MDPs and The Catastrophic Convergence Conjecture.

Original sequence description

Instrumental convergence posits that smart goal-directed agents will tend to take certain actions (e.g. gain resources, stay alive) in order to achieve their goals. These actions seem to involve taking power from humans. Human disempowerment seems like a key part of how AI might go very, very wrong.

But where does instrumental convergence come from? When does it occur, and how strongly? And what does the math look like?

Reward is not the optimization target

Many posts in this sequence treat reward functions as “specifying goals”, in some sense. This is wrong, as I have argued at length. Reward signals are akin to a per-datapoint learning rate. Reward chisels circuits into the AI. That’s it!

  1. Power as Easily Exploitable Opportunities
  2. Generalizing POWER to Multi-Agent Games
  3. MDP Models Are Determined by the Agent Architecture and the Environment
  4. Environmental Structure Can Cause Instrumental Convergence
  5. A World in Which the Alignment Problem Seems Lower-Stakes
  6. The More Power at Stake, the Stronger Instrumental Convergence
  7. Seeking Power Is Convergently Instrumental in a Broad Class of Environments
  8. When Most VNM-Coherent Preference Orderings Have Convergent Instrumental Incentives
  9. Satisficers Tend to Seek Power: Instrumental Convergence via Retargetability
  10. Instrumental Convergence for Realistic Agent Objectives
  11. Parametrically Retargetable Decision-Makers Tend to Seek Power

Original sequence description

My writings on different kinds of corrigibility. These thoughts build on each other and form part of my alignment worldview (circa 2021), but they are not yet woven into a coherent narrative.

  1. Non-Obstruction: A Simple Concept Motivating Corrigibility
  2. Corrigibility As Outside View
  3. A Certain Formalization of Corrigibility is VNM-Incoherent
  4. Formalizing Policy Modification Corrigibility

In early 2022, Quintin Pope and I noticed glaring problems at the heart of “classical” alignment arguments. We thought through the problem with fresh eyes and derived shard theory.

Classical arguments focus on what the goal of an AI will be. Why? There’s not a good answer that I’ve ever heard. Shard theory redirects our attention from fixed single objectives. The basic upshot of shard theory: AIs and humans are well-understood1 as having a bunch of situationally activated goals—“shards” of desire and preference.

For example, you probably care more about people you can see. Shard theory predicts this outcome. Consider your learned decision-making circuits which bid for actions which care for your friend Bill. These circuits were probably formed when you were able to see Bill (or perhaps the vast majority of your “caring about people” circuits were formed when physically around people). If you can see Bill, that situation is more “similar to the training distribution” for your “caring about Bill” shard. Therefore, the Bill shard is especially likely to fire when you can see him.

Thus, it seems OK if our AIs don’t have “perfect” shard mixtures. The stronger their “aligned shards”, the more human welfare weighs on their decision-making. We’re playing a game of inches, so let’s play to win.

  1. Humans Provide an Untapped Wealth of Evidence About Alignment
  2. Human Values & Biases Are Inaccessible to the Genome
  3. General Alignment Properties
  4. Evolution Is a Bad Analogy for AGI: Inner Alignment
  5. Reward Is Not the Optimization Target
  6. The Shard Theory of Human Values
  7. Understanding and Avoiding Value Drift
  8. A Shot at the Diamond-Alignment Problem
  9. Don’t Design Agents Which Exploit Adversarial Inputs
  10. Don’t Align Agents to Evaluations of Plans
  11. Alignment Allows “Nonrobust” Decision-Influences and Doesn’t Require Robust Grading
  12. Inner and Outer Alignment Decompose One Hard Problem Into Two Extremely Hard Problems

My work with my MATS 3.0 scholars, Ulisse Mini and Peli Grietzer!

A mouse with cheese subshards.
Original sequence description

Mechanistic interpretability on a pretrained policy network from Goal Misgeneralization in Deep Reinforcement Learning.

  1. Predictions for Shard Theory Mechanistic Interpretability Results
  2. Understanding and Controlling a Maze-Solving Policy Network
  3. Maze-Solving Agents: Add a Top-Right Vector, Make the Agent Go to the Top-Right
  4. Behavioral Statistics for a Maze-Solving Agent
☙Black and white trout❧

  1. Team Shard’s later experimental work provided strong evidence that AI policies are not just well-understood as having shards, but in fact mechanistically learn a blend of multiple situationally activated goals. ⤴