An outside evaluator can test a finished model. But what could they see if they worked close enough to follow training decisions as they happen? Anthropic’s 18 September partnership announcement says Accenture’s specialist AI business, Faculty, will do exactly that through an embedded evaluation arrangement. The ambition is deeper independent scrutiny of frontier models. The terms of independence and public reporting remain important open questions.
What Anthropic's embedded evaluation partnership covers
Anthropic says Faculty will evaluate and red-team models, assess alignment and test safeguards. It expects embedded evaluators to have access comparable to employees, so they can observe models as they take shape, follow decisions about building and deployment, and speak directly with staff. Both Anthropic and Accenture expect to invest at least $1 billion each in capacity for this area over five years. Those are stated expectations for a wider capacity build, not a disclosed fee for a single audit.
| Announced element | What it could enable | What is still unspecified |
|---|---|---|
| Employee-comparable access | Observation of training and deployment decisions earlier than a final model test. | Exact systems, records and decision rights available to evaluators. |
| Evaluation, red-teaming and safeguard testing | More chances to catch dangerous capability or weak controls. | Specific test plans and pass criteria. |
| Incident reporting and public account | More informed external scrutiny. | What must be published and when. |
| Direct Anthropic funding | The work can start before pooled funding exists. | How financial independence will be protected in practice. |
The announcement is unusually candid that embedded evaluation is new. Anthropic says there are no settled standards yet for evaluator access, reporting or funding. It is also talking with METR and other non-profit evaluators about pilots using their own funding. This Accenture partnership is non-exclusive on both sides.
Why access during development could change the assessment
A conventional external test often starts with a model that is close to release. Evaluators receive a defined interface, run agreed prompts or tasks, and report what they observe. That can reveal important weaknesses. It may not show why a safeguard was chosen, what a training run looked like before the final tuning, or whether a concerning pattern was seen earlier and judged acceptable.
The embedded approach aims to move the evaluator closer to those decisions. Anthropic says evaluators should be able to follow training, inspect decisions governing deployment and speak directly to employees. In principle, that creates a chance to ask better questions earlier: What risk was this safeguard designed to reduce? What evidence would cause a release to be delayed? Has the team tested how a model behaves with the tools it will actually use? The partnership announcement does not yet say that Accenture will have authority to stop a release, so readers should not infer a veto.
There is also a difference between access and influence. An evaluator can see a problem, yet the value of the arrangement depends on whether it can document the finding, challenge the company's interpretation and reach someone who can act. Those governance details will be as important as the volume of model testing.
Can embedded AI evaluation be independent?
It can, but the label alone is insufficient. Deeper access may let an evaluator see problems that a standard external benchmark misses. At the same time, the evaluated company controls much of the environment and, in this arrangement, directly funds the work. Anthropic says it remains responsible for its models. The practical test is whether evaluators can choose methods, record findings, escalate concerns and publish meaningful results even when those findings are uncomfortable.
| Question to ask of an embedded evaluation | Why it matters |
|---|---|
| Who defines the scope and can change it? | Narrow scope can exclude the riskiest behaviour. |
| Can evaluators inspect relevant training and tool-use traces? | A final score may hide how an agent reached an answer. |
| How are disagreements escalated? | Findings need a route beyond the project team. |
| Which results will be public? | Public value depends on more than a private assurance letter. |
This is an editorial assessment of the governance questions, not a claim that the partnership has failed any of them. Details are still being developed. For organisations purchasing AI, it is a reminder to ask what an “independent evaluation” actually examined and what access the evaluator had.
Four ways to test evaluator independence
Financial separation is one piece, but not the whole puzzle. Anthropic says it will directly fund Accenture's work because a pooled or public funding mechanism does not yet exist. That creates an obvious question about incentives, which the announcement itself acknowledges. A useful evaluation mandate should make the answers observable rather than relying on the evaluator's reputation alone.
First is methodological control: can the evaluation team choose additional tests when it sees an unexpected capability, or is it confined to a fixed checklist? Second is access: can it inspect logs, training decisions and safety incidents relevant to its conclusions? Third is escalation: can it raise a serious disagreement outside the team being assessed? Fourth is reporting: what can it tell customers, regulators or the public, and who gets to edit that account? These are criteria for judging future disclosures, not assertions about contractual terms that have not been published.
| Independence dimension | Stronger evidence to look for |
|---|---|
| Test design | Evaluators can pursue findings beyond a lab-selected prompt list. |
| Information access | Relevant model versions, traces and decisions are available for inspection. |
| Escalation | Material concerns reach decision-makers outside the evaluated project. |
| Reporting | Findings, limitations and unresolved disagreements can be communicated meaningfully. |
The best outcome would make it possible to distinguish what was tested, what was observed and what remains uncertain. A statement that a model was “independently reviewed” is not enough to answer those questions.
What embedded evaluation could change for frontier AI
Frontier model evaluations have often looked like short external tests of a largely finished system. Embedded work aims to inspect decisions across the development lifecycle. If it produces clear reporting standards and useful access, it could make safety commitments more verifiable. If the output stays private or narrowly scoped, the public will struggle to judge its value. The next meaningful milestone is a description of the evaluator’s mandate, evidence access and published findings.
What this means for enterprise buyers
Businesses rarely choose a frontier model on safety evidence alone. They also consider price, performance, data handling and integration. But deeper evaluations could give procurement teams better questions and better evidence. Ask whether the assessment covered the model version being bought, its planned tools, the specific business use case and any known limitations. A finding from a chat-only test may not transfer to an agent allowed to browse, run code or write to customer systems.
An enterprise buyer can also ask how an issue found after purchase will be handled. Will the vendor notify customers about material safety findings? Is there a release history for mitigations? Can the buyer pause or roll back a model update? Embedded evaluation may improve the vendor's evidence, but the customer still needs its own application-level tests for the permissions and workflows it controls.
The partnership is therefore best read as the beginning of an experiment in oversight. Anthropic has made a public commitment to bring a named evaluator closer to frontier development and has described the unresolved questions plainly. The proof will be in the access granted, the uncomfortable findings that can be raised and the information eventually made available outside the lab.
Frequently asked questions
Has Accenture already certified Anthropic’s models as safe?
No such certification appears in the announcement. It describes a partnership and intended evaluation work.
Is the $1 billion figure an audit contract?
No. Anthropic says each company expects to invest at least $1 billion in building capacity in this area over five years.
Will evaluation results be public?
Anthropic says embedded evaluators can report incidents and give the public a more informed account, but detailed reporting standards have not been settled.
