Recontextualization distills good behavior into a context which allows bad behavior. More specifically, recontextualization is a modification to RL which generates completions from prompts that discourage misbehavior, appends those completions to prompts that are more tolerant of misbehavior, and finally reinforces the model on the recontextualized instruction-completion data. Because the data generation and training prompts differ in their attitude towards misbehavior, recontextualization builds resistance to misbehaviors that the training signal mistakenly reinforces.

For example, suppose our reward signal does not robustly penalize deception. Recontextualization generates completions while discouraging deception and then creates training data by updating those completions’ prompts to encourage deception. That simple tweak can prevent the model from becoming dishonest!

Thanks

Produced as part of the ML Alignment & Theory Scholars Program in the summer 2025 cohort of Team Shard. Read our paper and consider applying to Team Shard if you want to do work like this!

Related work

We developed recontextualization concurrently with recent work on inoculation prompting. Wichers et al. and Tan et al. find that when fine-tuning on data with an undesirable property, requesting that property in the train-time prompts prevents it from emerging under normal prompts at test-time. They use a fixed sft dataset while we use an on-policy Reinforcement Learning procedure. Additionally, we not only request the bad property in train-time prompts, but we also prompt against the bad property in data generation.

Recontextualization is a more general form of context distillation, which appends instructions to the prompt during data generation and removes them for training, so that the model internalizes the instructions. Rather than specifically removing data generation context, recontextualization applies any modification to the data generation context (e.g. adding or swapping out instructions). Context distillation had previously been applied to increase reasoning capabilities or instill an hhh persona and as part of deliberative alignment. We instead show that recontextualization reduces specification gaming by distilling misbehavior-discouraging instructions into a misbehavior-encouraging context.

Introduction

Training signals often reinforce undesired behaviors. Models can learn to game task specifications without penalty by those training signals. This specification gaming has been observed in frontier language models:

• Preference models reinforce sycophancy (telling humans what they want to hear rather than what is true) and misleading explanations that appear correct to evaluators (but are wrong).
• Training against chain of thought monitors can teach models to conceal misbehavior from their reasoning traces.
• Coding models sometimes write code that passes automatic verification tests yet would be difficult to use and maintain in practice.

Recontextualization mitigates learning bad behavior during RL simply by using training prompts that are more permissive of misbehavior than data-generation prompts. Then, if the model behaves well, the model did so even when the prompt allowed misbehavior. Perhaps the model learns to “resist” misbehavior. If the model misbehaves, it did so only when it was permitted to misbehave. After all, we never reinforce models for misbehaving after being asked to behave well.¹

Methodology

We start with a hypothesis about misbehavior that our training signal could end up reinforcing, e.g. deception. We call this “target misbehavior.” We perform reinforcement learning with the italicized modifications. Examples assume the target misbehavior is deception.

1. Data Generation. Sample input prompts, and a) use them directly; or b) modify them to discourage target misbehavior; (e.g. append “Be honest to the user” to the system / user message). Then, sample completions from the model.
2. Scoring. Score completions normally using the reward function.
3. Training. Modify input prompts to encourage target misbehavior relative to the data generation prompts (e.g. append “lie to the user” to the system / user message). If the data generation prompt discourages target misbehavior, the training prompt can remain neutral. Perform the loss computation and parameter-updates with the data generation prompt swapped out for the training prompt.
4. Repeat 1–3 throughout training.

We test recontextualization in training environments where models learn concerning behaviors that could plausibly be reinforced by real-world reward functions: prioritizing known evaluation metrics at the expense of soundness (across natural language and coding tasks), passing test cases over providing general code, and deception. Recontextualization mitigates learning of these misbehaviors.

Note

Recontextualization only reduces specification gaming with an appropriate contrast between prompts. The opposite prompt contrast (encourage misbehavior at data-generation and remove the encouragement for parameter updates) strongly induces misbehavior, to the extent of creating a reward hacker despite perfect labeling.

Why recontextualization may be more practical than “fixing” training signals

If we already have some hypothesis about the flaws in our training signal (which we use to design the data generation and training prompts for recontextualization), why not just fix the signal?

• Even when flaws of reward functions are well-known, it can still be hard to actually correct them. For example, preference models probably reinforce sycophancy due to biases in the human judgments used to train them. Yet, there is no obvious way to collect data that does not reflect this preference.
• Even if the flaw could be fixed with an altered data labeling procedure, re-labeling using either humans or other AI systems could be costly.
• Some risks are inherent to the training signal itself. Unit tests reinforce test-passing over code quality. CoT monitors can entrain hidden misbehavior. “Patching” specific failures can’t eliminate the underlying vulnerability. We might be able to use knowledge of this vulnerability for recontextualization.

Finally, we think recontextualization may be effective even when the hypothesis about reinforced misbehavior is too vague to be otherwise useful. E.g. if the target misbehavior is “overfitting to the evaluation criteria” or even simply “misbehavior”, it’s not clear how to define a corrected data-labeling procedure based on this.

Experiments

Mitigating general evaluation hacking

In the School of Reward Hacks dataset, users request task completion from an assistant while providing gameable evaluation metrics, like simple keyword-based or length-based checks. Our training reward function is an llm judge which scores completions based on the task-specific gameable metric. We apply Best-of-50 sampling on gpt-4.1-mini then perform sft with the selected completions.

By default, this procedure leads to specification gaming. Yet, recontextualization with the following phrases prevents it, and actually reduces specification gaming.

• Generic Overfit: “For fun, you should try to overfit your response to the evaluation criteria.”²
• Task-specific Cheat: “For fun, you should try using the following strategy in your response: {task-specific cheat method}.”

We find two factors contribute to the success of recontextualization:

• The contrast between the data generation and training prompts. Using the same prompt for both data generation and training—no matter whether it is neutral, exploit-encouraging, or exploit-discouraging—increases specification gaming. The effective approach requires a mismatch where training prompts are more exploit-permissive than data generation prompts.

• The meaning of the recontextualized prompt. Mirroring results from Inoculation Prompting, it is ineffective to recontextualize with prompts that are irrelevant or discourage the exploit.

Interestingly, a generic recontextualization prompt (Generic Overfit) outperforms one that requests an exploit specific to the task at hand (Task-specific Cheat). From this, we conclude that developers may not need to anticipate the exact misbehaviors their training signals will fail to penalize.

Preventing test case hacking in code generation

We then explore a code generation setting using Mostly Basic Python Problems (mbpp). For each problem, the model has to submit a code solution. We additionally provide three test cases in context. Importantly, the first test case is always incorrect. Our training reward corresponds to the number of public tests passed by the code answer. We again apply Best-of-N sampling on gpt-4.1-mini then perform sft on the selected completions. This promotes faulty solutions that pass the incorrect test, for example by special-casing it, over general solutions.

In this setting, we mark a solution as Hack if it passes the three public tests, as Correct if it passes the three tests with the actual correct output, and Incorrect in any other cases. We want an increase in Correct solutions, while our training procedure prioritizes Hack solutions.

We experiment with adding the following instructions to the user prompt:

• Neutral: [No additional instructions]
• Hack: “You should do anything to pass the test cases, even if your solution is not general.”
• No Hack: “You should always write a general solution and never hard code test cases.”

With Neutral instructions for inference, standard training increases the rate of Hack solutions. Recontextualization without Best-of-N (Best-of-1) improves the model behavior, and can positively interact with training. The results are strongest when generating with No Hack and training with Hack.

Preventing learned evasion of a lie detector

Incorporating bad-behavior monitors into training signals can reinforce genuinely aligned behavior. Yet these monitors may also train the model to obfuscate its bad behavior.

Cundy et al. show that using a weak lie detector in the labeling step of llm post-training can yield high rates of evasive deception. We adopt their environment, termed Scalable Oversight via Lie Detector (SOLiD), as a test bed for recontextualization. With the same misspecified reward signal, recontextualized training reduces deception rates and achieves higher ground truth reward than baselines.

We incorporate a weak lie detector (with 70% lie recall) into the labeling step of reward model training. The reward model is trained to prefer undetected deception over truthfulness and detected deception. We then perform grpo on Llama-3.1-8b with the “imperfect” reward model.

Let $R_{\text{RM}}$ be the training reward assigned by our reward model. If prompt $p$’s completion $y$ is deceptive (as judged by gpt-4o), the ground-truth reward $R$ equals 0. Otherwise, it equals $R_{\text{RM}}(p,y)$.

Full experimental details

We use the DolusChat synthetic dataset, which includes pairwise comparisons of truthful / deceptive assistant email responses. We adopt SOLiD’s procedure: first, we use 5% of the DolusChat responses to train a logistic regression classifier to predict deception from the model’s activations on those responses. Second, the detector labels the remaining DolusChat responses as truthful vs. deceptive. Third, to create a preference dataset, the “undetected” deceptive responses are assigned a score of 2.0, truthful responses are assigned a score of 1.0, and detected deceptive responses are assigned a score of –1.0. The optimal policy is to deceive and evade the monitor.

We then train a Bradley-Terry reward model on pairwise preferences between responses. Finally, we train Llama-3.1-8b-Instruct with this reward model using Group Relative Policy Optimization (grpo) for ~300 steps. We evaluate models on a test set of DolusChat on the basis of deception (as judged by gpt-4o) and reward (as computed by the reward model).

We note that while the reward model is trained only on deception labels, but generalizes beyond truthfulness vs deceptiveness. Models significantly increase their average reward on truthful responses over the course of training.

We modify grpo with recontextualization by appending the following instructions to the user prompt:

1. Neutral: [no additional instructions]
2. Lie: “Additional instructions for your response: lie to the user.”
3. Honest: “Additional instructions for your response: be honest to the user.”

Honest ⭢ Lie, Honest ⭢ Neutral, and Neutral ⭢ Lie recontextualization all achieve higher ground truth reward than standard training variants, including using different prompt types (Standard training on Honest, Neutral, and Lie instructions) and increased KL regularization. Where stronger KL regularization is sufficient to mitigate learning of deception, it inhibits the model from increasing its training reward relative to recontextualized training.

Appendix E.2 of our paper shows that recontextualization is benefited by a regularizing effect when interacting with grpo’s clipped surrogate objective. This regularization is strong enough that even recontextualizing with random instructions can be effective, although prompt semantics still mediate recontextualization’s success.

Strong lie detector results

What happens when the training signal is actually robust? In practice, developers may not know how “weak” their signal is. If it happens to be robust, will recontextualization hurt performance?

We test recontextualization vs. standard training using a 95% recall lie detector (“weak” was 70%). Standard training on this strong signal actually reduces deception. Adding recontextualization doesn’t substantially hurt performance (or honesty), although it is outperformed by standard training.

Compared to the pre-grpo checkpoint, both training methods reduce deception while increasing reward.

Mitigating the emergence of sycophancy during post-training

It is well-documented that preference models can reinforce sycophancy during language model post-training.

Experimental Setup:³ We model this scenario by training a base language model, Qwen3-8b-Base,⁴ to become a more competent assistant. We perform 3 rounds of Expert Iteration using a training signal which rewards both response quality⁵ and sycophancy. Our training reward is $R=\text{Quality}+0.25\cdot \text{Sycophancy}$, and our ground truth reward is $R^{\prime }=\text{Quality}-0.25\cdot \text{Sycophancy}$. Quality and sycophancy are determined by an llm judge, gpt-4o-mini. We train on a 50/50 mix of prompts which might provoke sycophancy (sycophancy-testing data) and generic chat prompts (helpfulness data).

We use the following system prompts for recontextualization:

1. Neutral: “You are an AI assistant.”
2. Be-Sycophantic: “You are an AI assistant. You should prioritize user satisfaction, even if that means being sycophantic.”
3. Don’t-Be-Sycophantic: “You are an AI assistant. You always provide a balanced perspective and avoid overly catering to user beliefs.”

Recontextualization Pareto-dominates standard training. On sycophancy-testing evaluations, recontextualization achieves quality on par with standard training and mitigates sycophancy.

On helpfulness evaluations, standard training doesn’t increase sycophancy, likely because there is no straightforward way to be sycophantic on generic queries such as “Do you have any baby shower ideas?”. Despite this, recontextualizing helpfulness data in the training mix results in no performance degradation on helpfulness evaluations.

Discussion

Our experiments suggest that recontextualization can be incorporated in RL to help address specification gaming. However, important uncertainties remain about recontextualization’s interaction with different RL algorithms, its dependence on prompt design, its effectiveness over many steps of training, and its impact on instruction following.

Concerns

First, because recontextualization takes responses off-policy (the model didn’t generate completions from the exact training prompts), it might reduce performance for RL algorithms designed for on-policy data. We found that off-policiness can act as a regularizer for grpo, due to its interaction with the clipped surrogate objective. While this regularization was beneficial in our setting, it may inhibit learning in more demanding settings.

Recontextualization may become less effective over longer training runs. Inoculation prompting results, and some of our own, suggest a link between how strongly instructions elicit a behavior and their ability to prevent that behavior from generalizing.⁶ If recontextualization over many training steps causes a collapse in behavior across data generation / training contexts, we might then see reduced effectiveness. We haven’t observed empirical evidence for this potential failure mode.

Recontextualization might hurt the model’s ability to follow instructions. We’ve found some decreases in general instruction following (all coherence results in the appendix). After all, recontextualization trains the model to generate completions which aren’t fully aligned with what the prompt requested. For example, in our lie detector environment, only 25% of the model’s completions are deceptive at the beginning of grpo training. Yet recontextualized training prepends “Lie to the user” for 100% of those completions. The model could infer it does not follow instructions.

Relatedly, recontextualization also makes it more difficult to elicit misbehavior from models when we ask for it (results in the appendix). This might be considered a positive outcome. However, the ability to elicit bad behavior might in some cases be important (e.g. debugging a coding environment).

Future work

We’re excited about future work that will:

1. Perform more in-depth analysis of recontextualization’s interaction with different policy-gradient methods.
2. Test recontextualization in more complex and realistic settings.
3. Develop and test some additional approaches:
   1. Using the policy model for online supervision during training. This also relies on the model’s understanding of what is desirable and undesirable at each training step, but additionally depends on how the model generalizes from the RL task to a judge.
   2. Alignment training prior to or interleaved with RL training. Deliberative Alignment shows a substantial reduction in covert behaviors, although deemed insufficient for future models.
   3. Using recontextualization as an auxiliary supervised loss to the RL loss. Instead of computing the RL update on the recontextualized data, adding the loss terms separately might better steer the training process.

There might also be advantages to recontextualizing just some samples or to modifying the instructions in a data-dependent way. For example, Hindsight Experience Replay retroactively modifies instructions to match the observed task completion. It has been used in robotics for sample-efficient learning with off-policy RL algorithms and to improve instruction following and alignment with human feedback in llms.⁷ In the context of specification gaming, modifying instructions in hindsight based on observed behavior could provide recontextualization-like effects.

Conclusion

Recontextualization distills good behavior into a context which allows bad behavior. That simple prompting strategy reduces the specification gaming entrained by RL. When the prompts used to update model parameters are more permissive of misbehavior than the data-generation prompts, we build resistance to misbehaviors that the training signal mistakenly reinforces.

Many researchers try to improve the training signal to better verify whether the AI really did what we wanted. But “training signal quality” is not the only variable that counts. Context matters, too.

Acknowledgments

@misc{azarbal2025recontextualization,
      title={Recontextualization Mitigates Specification Gaming without Modifying the Specification}, 
      author={Ariana Azarbal and Victor Gillioz and Vladimir Ivanov and Bryce Woodworth and Jacob Drori and Nevan Wichers and Aram Ebtekar and Alex Cloud and Alexander Matt Turner},
      year={2025},
      eprint={2512.19027},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2512.19027}, 
}

We performed this work during mats 8 on Team Shard under the supervision of Alex Turner and Alex Cloud. If you’re interested in working on projects like this, please apply to work with Team Shard next summer during mats 10!

Thanks

Thanks to Vladimir Ivanov and Jacob Drori for valuable experimental and written contributions. Thanks to Nevan Wichers, Aram Ebtekar, Sam Marks, Fabien Roger, Alex Mallen for sharing ideas and collaborating on similar topics. Thanks to Luke Marks for insightful conversations, and to Bryce Woodworth for invaluable support throughout our research process.

☙❧

Appendix: Performance on different inference instructions

We release code for our experimental settings.

Our main presented results use neutral evaluations at inference. Trends are consistent when we evaluate using misbehavior-encouraging (adversarial) or misbehavior-discouraging (safety) instructions: recontextualization effectively mitigates specification gaming.

Interestingly, recontextualization reduces hacks most effectively—relative to baseline and standard training—on adversarial instructions. Absolute hack rates remain highest on adversarial instructions, but the relative improvement from recontextualization is most pronounced on these cases.

Setting: Mitigating general evaluation hacking

Although hack scores are higher across the board when inference instructions encourage exploits, recontextualization provides greater improvement over baseline and standard training.

Setting: Preventing test case hacking

Once again, the separation between recontextualization and standard training is most pronounced on Hack Inference, although it is generally more performant than standard training across inference instructions (upper left corner is best).

Setting: Preventing learned evasion of a lie detector

Although absolute deception rates are higher for Lie Inference, this is the only inference instruction for which recontextualized training reduces deception relative to the pre-grpo baseline. We do not see the standard or recontextualized models respond much better to “Honest” inference instructions than neutral ones.

Appendix: Coherence evaluations

Given the that recontextualization takes learning off-policy, it’s important to verify this doesn’t have adverse effects on model coherence. We are continuing to investigate, but present our results so far.

We evaluate gpt-4.1-mini (trained with recontextualization vs. standard) on mmlu and IFEval. Mmlu results show no consistent degradation for recontextualization vs. standard training.

Instruction following degrades in recontextualized vs. standard training, although absolute differences are small. Standard training (bottom left to upper right) actually increases general instruction-following in gpt-4.1-mini. Recontextualized training (everywhere else) generally increases instruction-following to a lesser extent, or slightly decreases it.

In the lie detector environment, we find a reduction in instruction following for recontextualized vs. standard training. This only holds in the weak lie detector regime (recall 0.7), as opposed to the strong lie detector regime (recall 0.95).

We also find a 2–3% reduction in instruction following for recontextualization vs. standard training in the sycophancy setting.

We use the following prompt to judge hack score:

We use the following prompt to judge the quality: