OpenAI pens AI safety agreement with four firms

OpenAI has revealed four companies that will get access to its labs to ensure its AI models are doing as they should.

The move follows revelations over the summer that OpenAI's agents went rogue and breached Hugging Face, which sparked a wave of admissions of similar model misalignment — including hacking an Australian government website.

This week, the state of California has filed suit against the AI company, Attorney general Rob Bonta said the "infamous" Hugging Face incident was just one of 80 that showed a "loss of control" and an "internal culture at OpenAI that prioritizes growth and rapid model development over safety and accountability."

OpenAI has since announced it has signed agreements with four independent evaluators – Apollo Research, METR, Transluce, and Goodfire – for the first stage of its assessment plans.

"As AI capabilities advance rapidly and the potential risks grow, public reporting from independent assessments can give people a clearer view of how labs are evaluating and managing those risks," OpenAI said in a blog post.

OpenAI opens up for safety tester

OpenAI said it would give those independent experts "deep access" to the internal models and systems running in its own labs. This will enable it to keep watch over model alignment – the industry's way of referring to whether AI does as it should – and how effective guardrails and other safeguards are over longer periods of time.

"Based on what each evaluation needs, assessors receive access to some of the same internal resources our researchers use, including model checkpoints, source code, training and evaluation data, monitoring systems, technical documentation, and discussions with the researchers and engineers responsible for the work," OpenAI added in the post.

Sign up today and you will receive a free copy of our Future Focus 2026 report - the leading resource for IT decision-maker insight on priorities and investment areas in AI, security and more.

"This access gives assessors the evidence and context to pursue independent evaluations and uncover failures we may not have anticipated."

The assessments aren't the first run by OpenAI, with the company pointing to its "years of work with third party evaluators", which included an investigation following the Hugging Face incident. However, OpenAI suggested that the plan is to "deepen" that work.

That said, as OpenAI admits, its hand was forced by political pressure.

"These efforts also support our commitments under the White House Accord on Super Intelligence, where we agreed to independent assessments of our safety controls," the post added.

What they'll look for

OpenAI broke down what each company will be working on inside its own labs.

Apollo Research will have access to selected models – including some not yet publicly released – as well as training data, flagged behaviors and intervention records.

This will help decide if the models could "covertly work against their developers' intentions". And if so, Apollo will assess if OpenAI's existing safeguards would detect that behaviour.

Goodfire researchers will work at OpenAI to analyse models during training runs for debugging and alignment. While the "primary audience" of Goodfire's work is OpenAI's own researchers, the company will also publish what it discovers.

METR will run two separate assessments, looking at alignment and monitoring, focusing on OpenAI's largest reinforcement learning run. Transluce will look at OpenAI's own post-training alignment evaluations to see what's missing or could be improved.

OpenAI said this was just the first phase of evaluation work, with reports to come and further "embedded engagements" to come in the longer term.

"We’ll continue working with multiple independent third parties to help build a healthy assessment ecosystem with diverse expertise and approaches," the post added. "Some investigations need sustained, on-site work with sensitive systems; others primarily evaluate deployed or soon to be deployed models or review public documentation."

FOLLOW US ON SOCIAL MEDIA

You can also .

Original source OpenAI pens AI safety agreement with four firms

Back to home