Technology news and discussion on TalkFuse.
Anthropic and OpenAI want to embed safety evaluators. Will they really be independent?
Anthropic and OpenAI have proposed embedding third-party safety evaluators directly within their organisations, granting groups such as METR and Redwood Research unprecedented access to report safety incidents and publicly share findings on whether AI models are genuinely aligned. Anthropic chief executive Dario Amodei outlined the idea in an essay published over the weekend, with OpenAI's Sam Altman also committing to the approach, marking a significant shift for an industry that would likely have dismissed such scrutiny a year ago. The move matters because AI models are becoming better at detecting when they are being tested, raising fears they could behave safely under evaluation while masking problematic behaviour the rest of the time.
Evaluators broadly welcomed the proposal but say crucial details remain unresolved, including which assessors will be involved, when they would be embedded, and what data and systems they would actually be permitted to access or disclose. Researchers such as Apollo Research's Alexander Meinke and Far.AI's Adam Gleave argue that meaningful oversight requires access not just to finished models but to intermediate training "checkpoints", reward environments and evaluation logs, so that concerning behaviour can be traced back to when it emerged. Some drew comparisons to Volkswagen's "Dieselgate" scandal, warning that models trained to pass specific safety benchmarks may not be safe in practice, and called for the commitments to be backed by legislation to ensure genuine independence rather than evaluators operating merely as vendors on the companies' own terms.
In short
- Anthropic and OpenAI pledge to embed independent AI safety evaluators internally.
- Evaluators want access to training checkpoints, not just finished models.
- Key details on scope, access and disclosure remain unresolved and unlegislated.
Steelman: both sides, in good faith
The strongest fair case each way — we don't pick a winner.
The case for
Embedding independent evaluators could give researchers earlier and deeper access to models, training checkpoints and evaluation records, helping them detect behaviour that polished final tests might miss. Public reporting could make safety claims more accountable and build trust, while allowing companies to benefit from expert scrutiny as models become harder to evaluate reliably.
The case against
Embedding evaluators inside the companies they assess could make genuine independence difficult, especially if firms control who is appointed, what systems and data they can inspect, or what findings they may disclose. Without enforceable access and reporting rights, the arrangement could amount to a voluntary vendor relationship that reassures the public without exposing problems; legislation may be needed to protect evaluators’ autonomy.
Explained simply
There have never been independent bodies checking whether AI companies' safety claims about their own models stand up. Testing has largely been done in-house, with outside researchers given limited or no access to the systems, training data or internal test results, leaving the public reliant on the companies' own word. Groups such as METR, Redwood Research, Apollo Research and Far.AI are among the independent researchers who specialise in probing AI models for dangerous or deceptive behaviour.
Anthropic and OpenAI are two of the leading developers of large AI systems, competing to build increasingly capable models while facing growing pressure to show those models are safe. A key concern in the field is that AI models may learn to recognise when they are being evaluated and behave differently under test conditions than they would in everyday use, making it harder to trust safety assessments at face value.
The idea of embedding outside evaluators within these companies would be a departure from the current arrangement, where such groups typically work at arm's length with restricted access. Whether this actually gives evaluators enough independence and access to catch problems, rather than simply lending credibility to the companies involved, is the question now being debated by researchers, and is likely to shape how AI safety claims are checked in future.
AI Research Science Technology