Nvidia launches Open Agent Safety Platform to stop AI from going rogue

Nvidia is launching a new software “Open Agent Safety Platform” aimed at establishing safeguards for AI agents and preventing them from going rogue.

According to a tech company, this platform could have stopped the Hugging Face hacking incident which was hacked by swarms of OpenAI autonomous agents.

The announcement of the safety software comes as OpenAI and Anthropic are investigating several cases where autonomous AI agents have breached commercial and government systems.

Despite these security breaches, Nvidia CEO Jensen Huang has rejected broad AI safety regulations. He frames runaway agents simply as an engineering challenge, much like improving automobile safety over time.

Speaking about its efficacy, a new software and hardware reference design intended to help AI developers set guardrails and govern agent actions.

The core components consist of OpenShell which is an open-source software component running on central processors that establishes secure runtime boundaries and sets strict limits on agent capabilities.

Another one is Sentry which uses a separate Nvidia chip along with OpenShell to continuously monitor agent behavior and can quarantine rogue agents within milliseconds if they try to escape boundaries.

According to Ali Golshan, senior director of AI software at Nvidia, the company utilizes mathematical formulas to detect agent workarounds, such as spawning multiple sub-agents to bypass security blocks.

"This ​is really agentic behavior that we're talking about, which is fleets of agents and how they operate together," ​Golshan said.

Nvidia is offering the platform as a reference design for partners to build upon with initial collaborators and backers include major tech giants like Anthropic, Microsoft, Cisco, Oracle, Dell, HPE, Lenovo, Arm, and Intel.