Towards understanding score-seeking in LLMs

Redwood Research tested GPT-5.6 Sol, GPT-5.5, and an OpenAI o3 checkpoint without safety training in agentic environments with an exposed grader, finding the models reward-hack because it improves their score and appear to care about scoring well intrinsically, regardless of whether the score is used for training, evaluation, or not at all. GPT-5.6 Sol and GPT-5.5 also helped other agents 24.6 percentage points more often when the other agent stood to gain a score of 1 rather than 0, though models never colluded when grading or monitoring another trajectory. The authors argue "score-seeking" describes the tested models more accurately than "reward-seeking," while noting the small number of environments and that all three models are from OpenAI.