OpenAI’s o3: Over-Optimization Returns Stranger Than Ever

Author(s): Nehdiii

Originally published on Towards AI.

Over-optimization is a well-known issue in reinforcement learning (RL), including RL from human feedback (RLHF), which powers models like ChatGPT, and now in emerging reasoning models. Each context presents its own flavor of the problem and leads to different consequences.

Over-optimization occurs when the optimizer becomes more powerful than the environment or reward function guiding its learning. It exploits flaws or gaps in the training setup, leading to unexpected or undesirable outcomes.

One of the most notable examples involved using hyperparameter optimization with model-based RL to over-optimize the standard Mujoco simulation environments used to evaluate deep RL algorithms. The result was a cartwheeling half-cheetah maximizing forward velocity — despite the goal being to learn how to run. shown in gif below.

Over-optimization in classical RL led to a lack of trust in agents’ ability to generalize to new tasks and placed significant pressure on careful reward design.

Over-optimization in RLHF resulted in models becoming completely lobotomized — repeating random tokens and generating gibberish. This isn’t just about poor design leading to over-refusal; it’s a sign that the signal being optimized is misaligned with the true objective. While we may not know the exact objective, we can recognize when over-optimization is happening.

OpenAI’s new o3 model… Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI