Link: Misleading Metaphors and Real Risks via Melanie Mitchell

After OpenAI’s agents hacked into HuggingFace’s servers, Melanie Mitchell wrote a grounded explanation of what happened. She explains that the HuggingFace incident was due to poor testing conditions and reinforcement learning methods that incentivized “reward hacking” and message passing.

First, OpenAI did not have proper security measures in place. They turned off safeguards built into the models, instructed the models to find and exploit software vulnerabilities, and let the models run autonomously for weeks without sufficient human oversight. OpenAI trusted the sandbox software to keep things safe, even though the company had previously seen similar “high persistence” models trying to hack out of sandbox software.

Gary McGraw, CEO of the Berryville Institute of Machine Learning, told me, “Building a sandbox that cannot be escaped has been a known and well-studied security problem since the early days of Java applets… Unfortunately, OpenAI appears not to have taken many of the lessons that malware analysts learned the hard way to heart.” And Niels Provos, former head of security at Stripe, was quoted in Wired as saying, “This should not have happened… I wish the frontier labs spent as much time on teaching their models to write secure infrastructure as they are spending on them exploiting vulnerabilities.”

The second factor includes a phenomenon that has been recognized for decades: when AI systems are given a problem to solve, they will sometimes find “shortcuts”—tricks for getting answers in ways that were not intended by the humans who posed the problem.

In RL an agent is given a task and possible actions it can take to accomplish the task, and earns a reward for successfully completing the task. For example, a hypothetical hacking challenge might consist of giving an agent a computer program with a known vulnerability and challenging the agent to use that vulnerability to access a forbidden file that contains a secret word. The agent then sets about taking many actions, for example, generating and running code, using a debugger to fix code, or renaming computer files. The agent receives a “reward” (a high number) only if the task is successfully solved—for example, the agent shows that it has the secret word. This training is repeated many times, and successful strategies are reinforced when rewards are gained.

The history of reinforcement learning is replete with examples of agents using shortcuts—in this case, called “reward hacking.” In one famous example, a programmer used RL to train his Roomba robot vacuum cleaner to go faster while avoiding bumping into things. The robot obtained rewards when it moved quickly and avoided triggering the robot’s front bumper sensors. The result: the Roomba figured out that that it could obtain rewards not by actually learning to avoid bumping into things, but by driving backwards, since its back bumper did not have any sensors. In short, reward hacking is when an agent can obtain rewards in a way that is easier than actually performing its task in the way humans intended

This kind of reward hacking, combined with very long training periods that encourage extraordinary persistence, is what happened with OpenAI’s agents. The company ran largely unmonitored reinforcement learning sessions over many weeks or months, training agents to autonomously perform tasks that require many different steps, to be “highly persistent” in trying many possible solution paths, and to get “creative” when their attempts fail. The agents were rewarded when they showed they had obtained a solution, no matter how that solution had been reached. According to OpenAI, the same kind of reward-hacking behavior seen in the Hugging Face hacking was also occurring during training, and this behavior was reinforced by the training algorithm:

Moreover, the fact that agents figured out how to communicate with one another—a result that so surprised OpenAI engineers—was also likely due to intentional or unintentional reinforcement when the underlying model was being trained: “We found…that these severe forms of unauthorized communication (e.g., sharing data with other agents) rose over the course of a training run that contributed to [the model’s subsequent behavior].” “Agent communication is not problematic behavior on its own—we train and deploy multi-agent systems that can communicate on the same task via a collaboration tool…. We believe that [the unintended agent communication strategies] started due to generalization from multi-agent training.” In short, OpenAI learned that the hacking and message-passing behaviors exhibited in the evaluation of its agents had all first appeared during training runs, which reinforced such behavior.

In short, OpenAI learned that the hacking and message-passing behaviors exhibited in the evaluation of its agents had all first appeared during training runs, which reinforced such behavior. In this particular incident, during the evaluation phase, the agents were given tasks that they could not solve via their own reasoning and coding abilities. So they attempted a shortcut—manipulating the way solutions were graded in order to get the reward they had been trained to relentlessly pursue.

While I agree with the general sentiments behind these lawmakers’ reactions—that humans should always remain in control of AI systems—it is essential for lawmakers, and the public, to understand that none of the reported incidents actually involved loss of control at any time, or arguably even “rogue agents,” or any kind of humanlike agency on the part of AI models. Instead, the blame lies with the humans who failed at engineering safe testing conditions, and who train AI models using RL methods that incentivize high persistence, autonomous decision-making, and reward hacking.

But perhaps most challenging of all, such proposals would require reimagining what AI is. It would require moving away from misleading anthropomorphic metaphors and narratives of inevitability, leaving behind “rogue AI agents” and embracing our own human agency to decide what we want AI to do for us. It might even require abandoning the seemingly hopeless project of “AI alignment”—teaching machines to be “good”—and embracing the idea that AI’s proper role is not as a moral agent, but as a set of tools to enhance human thriving.

Via: https://aiguide.substack.com/p/misleading-metaphors-and-real-risks