Frontier AI developers publish safety frameworks to explain how they assess dangerous capabilities and what they will do when those capabilities emerge. But a framework’s familiar name can outlast the commitments that gave it meaning. The question is not simply whether a company has a safety policy. It is whether outsiders can tell when that policy becomes less demanding.
A September 2026 study offers evidence of that problem. Among eight framework revisions accompanied by developers’ own explanations, it classified 67% of material changes as silent under a strict disclosure standard and 53% under a more permissive one. Changes that weakened existing commitments were more often inadequately disclosed than changes that strengthened them.[1]
Those findings warrant attention, but they also require care. The study uses language-model-assisted coding, lacks a completed independent reliability assessment, and distinguishes changes to existing commitments from entirely new commitments. It does not establish that three-quarters of all policy changes weaken safety, nor does it measure whether models themselves became more dangerous.[2]
The concern is narrower and more defensible: safety commitments can become less constraining without their published revision accounts making the loss of protection clear. Meanwhile, proposed legislation raises a separate question about whether external oversight will constrain that discretion or leave it substantially intact.
What the study measured
Louis Yiven Zhu’s “Silent Revision: Measuring Undisclosed Change in the Safety Frameworks of Frontier AI Developers” was submitted to arXiv on September 8, 2026. Its first version describes a corpus covering twelve developers, but only seven had multiple labeled versions and entered its longitudinal analysis. Twelve version pairs were traced at the commitment level, producing 710 commitment instances. The headline disclosure rates come from eight pairs with revision accounts, not from all twelve developers equally.[1]
A commitment was a statement describing something the developer undertook to do, at strengths ranging from “must” to “may.” A change counted as material if it altered scope, a threshold or trigger, the responsible actor, obligation strength, disclosure scope, or consequences. The analysis then asked whether a reader of the developer’s account alone would learn that the commitment had changed in the relevant direction.[2]
The strict measure counts partially announced changes as silent: naming a topic without describing the substance or direction of its revision is insufficient. The lenient measure counts those partial announcements as disclosed. Across the eight pairs, the respective counts were 257 and 203 out of 383 material changes.[1]
The direction-of-change finding has a different denominator. Of 299 material changes to existing commitments, 229—77%—were classified as weakening, removal, or relocation. That calculation excludes 174 additions. Including those additions in the denominator would make the weakening/removal/relocation category about 48% of all 473 material changes reported across the twelve traced pairs. Counting additions does not establish that they offset lost constraints, but omitting them would give an incomplete picture of the revisions.[3]
Among existing commitments in the eight pairs with accounts, the strict silence rate was 75% for weakenings, removals, or relocations, compared with 50% for strengthenings. The asymmetry appeared in seven of the eight pairs. That within-pair pattern is more informative than a claim that developers simply weaken everything: it suggests that unfavorable changes are less visible in the explanations readers receive.[3]
The evidence has important limits
The study’s first coding pass was produced by an agentic system from the Claude language-model family. The author individually adjudicated 244 of 710 rows and accepted the rest after a spot-check of 45 rows produced one change. Independent second-coder agreement was incomplete at submission. Checking that quotations match source documents establishes quotation accuracy, not the reliability of judgments about whether a commitment was weakened or adequately disclosed.[2]
Other coding choices matter. Changes that both strengthen and weaken a commitment are classified as weakened and flagged as mixed. Relocation includes movement into companion documents or nonbinding recommendations. These decisions are explicit, but they remain interpretive choices rather than self-evident facts.[2]
The disclosure rate also depends on granularity. Collapsing commitments into sections reduced the strict rate to 49% and the lenient rate to 34%. Redlined revisions received a zero silence rate by construction, although showing every textual edit is not necessarily the same as explaining its significance.[4]
These qualifications do not erase the observed pattern. They mean it should be presented as evidence from one systematic, inspectable analysis, not as a fully validated estimate of industry-wide behavior. The paper’s tables and discussion also contain inconsistencies that warrant clarification before its results are treated as definitive.[4]
Most importantly, the study measures written obligations and their visibility. It neither establishes deliberate concealment nor demonstrates deterioration in operational safety. A company could weaken one written commitment while improving a safeguard elsewhere. That possibility makes a complete revision record more important, not less.
Not every retreat is silent
Anthropic’s February 24, 2026 rewrite of its Responsible Scaling Policy illustrates both the substantive change and the distinction between announcement and enumeration.
The company publicly explained the revision. It described shortcomings in its earlier approach: capability thresholds were ambiguous, evaluation science could not reliably determine when some thresholds had been crossed, and safeguards for more advanced systems could require government assistance or collective action. Those are substantive reasons for revisiting a policy.[5]
The new approach also added reporting and review mechanisms. It introduced a Frontier Safety Roadmap, recurring Risk Reports, and external review under specified circumstances. Yet Anthropic explicitly described the roadmap’s goals as “nonbinding but publicly-declared” rather than hard commitments.[5]
Reporting progress toward a goal is different from meeting a condition before proceeding. A report may reveal a gap between capabilities and safeguards without requiring the gap to be closed before development continues.
Nor does a public explanation of that strategic shift necessarily identify every consequential removal. Zhu’s analysis classifies Anthropic’s transition from RSP 2.2 to 3.0 as a narrative-account revision and reports that 51 of its 86 material changes were not identified even under the lenient standard. Examples include removing a commitment to delete model weights when required security safeguards could not be met and replacing a training-pause commitment with language about reducing interim risk.[6]
The distinction matters. The overall rewrite was openly announced; the study argues that many individual changes were not adequately enumerated. Public retreat and silent revision can coexist within the same update.
A legal duty can still leave the standard movable
California’s Transparency in Frontier Artificial Intelligence Act goes beyond requiring publication. It requires a large frontier developer to “write, implement, comply with” its framework. Failure to comply with that framework can attract a civil penalty of up to $1 million per violation in an action brought by the attorney general.[7]
It would therefore be wrong to describe California’s framework regime as merely voluntary or purely informational. The state has made compliance with a developer’s framework a legal obligation.
But the framework remains revisable. The law requires annual review and, after a material modification, publication of the revised framework and a justification within thirty days. It does not expressly require a commitment-by-commitment redline or directional inventory of those changes.[7]
That creates the distinction at the center of this essay: enforcing compliance with a published framework is not the same as fixing every substantive obligation within that framework permanently in law. An explanation of why a policy changed can leave readers uncertain about precisely which constraints changed with it.
The question is therefore not whether these documents have any legal force. In California, they do. It is how the law governs changes to the obligations it helps enforce.
The federal proposal raises a separate concern
Anthropic’s proposed Artificial Intelligence Evaluation, Governance, and Incident Safety Act, or AEGIS Act, brings external authority into the discussion. The September 29 discussion draft described in October reporting was a proposal, not enacted law. Its provisions and the company’s promised revisions must be kept separate.[8]
Under the reported proposal, the Commerce Department could ask a federal court to block a model under conditions involving an imminent, foreseeable enumerated risk. The definition included more than 50 deaths or serious injuries, or more than $1 billion in damages, arising from specified categories of incidents.[9]
Anthropic defended the numerical threshold by noting its similarity to California law. But the same numbers can perform different legal functions. California uses its catastrophic-risk definition in a framework, assessment, and reporting regime; the federal proposal would also use a comparable threshold as a condition for its model-blocking mechanism. Borrowing a definition does not establish that the resulting intervention power is equally protective.7
Reporting on the draft also describes preemption of state requirements concerning safety reporting and testing. Anthropic’s head of federal affairs, Anthony Cimino, acknowledged concerns and said the company planned to revise its discussion language, including to clarify state roles concerning independent evaluators and a lab’s capacity to shut down a model. He also said the proposed blocking mechanism would not replace other government authorities or absolve companies of liability.[8]
Those qualifications prevent a sweeping claim that the proposal would eliminate every other route to intervention. They do not resolve the central trade: which state protections would disappear, and what federal protections would replace them?
A federal law could add useful oversight while weakening accountability in particular areas. Its strength must be judged provision by provision, not inferred from the existence of a national standard. The legislative discussion is therefore related to silent revision, but it is not another measurement of the same phenomenon.
What auditable revision would require
The immediate remedy is an enumeration duty. A developer should identify material changes to its commitments, not merely explain the purpose of a new version. Zhu’s paper makes that distinction explicit.[10]
An auditable process should provide:
- A permanent archive of dated versions.
- A redline showing exact textual changes.
- A plain-language inventory of commitments strengthened, weakened, removed, relocated, or newly qualified.
- An explanation of replacement protections and when the changes take effect.
Relocated commitments should be traceable into the documents where they survive. Security-sensitive information may require redaction, but that should not become a general excuse for hiding the direction of a policy change.
These are proposals for better accountability, not a description of requirements already uniformly in force. They also have a clear limit: making a retreat visible does not prevent it. Enumeration makes revision auditable; it does not establish a minimum level of safety.
Transparency and substantive constraints must therefore remain separate questions. One concerns whether outsiders can see what developers have decided. The other concerns which decisions developers should be free to make on their own.
The framework’s name is not the commitment
The available evidence supports concern about the visibility and durability of written safety obligations. It does not justify treating every revision as harmful, every omission as intentional, or every weakened clause as proof that an AI system has become less safe.
The stronger argument does not need those claims. A familiar policy name can survive the removal of an important condition. A detailed announcement can explain a strategy without identifying every lost protection. A legal regime can require compliance with a framework while leaving substantial discretion over its future terms.
For regulators, customers, and the public, the practical test is not simply whether a developer publishes a safety framework. It is what the current framework requires, how those requirements have changed, and what happens when the developer cannot meet them.
Unless those questions have clear answers, the document can retain the form of accountability while its substance erodes.
Notes
- Louis Yiven Zhu, “Silent Revision: Measuring Undisclosed Change in the Safety Frameworks of Frontier AI Developers,” arXiv:2609.08789v1, September 8, 2026, secs. 3–5, https://arxiv.org/html/2609.08789v1. The version reviewed states that it is under workshop review; it is not treated here as peer-reviewed validation.
- Zhu, “Silent Revision,” sec. 4 and sec. 7. The author reports Claude-family first-pass coding, individual adjudication of 244 rows, and incomplete independent second coding. Sampling parameters for the first-pass coder were not fixed. These limitations preclude claiming an independently replicated coding result.
- Zhu, “Silent Revision,” sec. 5 and table 1. The reported 77% is 229/299 and includes weakened, removed, and relocated existing commitments; it excludes additions. Table 1 reports 174 additions across the twelve pairs. Including them yields 229/473, approximately 48%. These are counts of coded changes, not weights for their relative safety importance. The 75% versus 50% visibility comparison uses the strict disclosure standard and 181 versus 50 changes in existing commitments with accounts.
- Zhu, “Silent Revision,” secs. 3–5 and 7. The paper reports 49% strict and 34% lenient silence at section granularity. It describes five redlined pairs as having zero silence by construction. However, its text says Anthropic supplied a redline for every revision from 2.2 onward while also identifying the 2.2-to-3.0 transition as a non-redlined exception. Its before/after discussion refers to “three pairs a side” after reporting three before and five after. Table 1’s eight with-account rows sum to 152 additions, whereas a later sentence calls Meta’s 57 additions and Anthropic 3.0’s 26 part of a “further 174”; 174 is the twelve-pair total. The essay uses the table and the explicit pooled denominators, does not rely on the ambiguous redline generalization, and makes no causal claim about the California law’s effect. This review checked the published method and internal arithmetic; it did not rerun the released coding pipeline or independently recode the corpus.
- Anthropic, “Anthropic’s Responsible Scaling Policy: Version 3.0,” February 24, 2026, https://www.anthropic.com/news/responsible-scaling-policy-v3. The description of roadmap goals, Risk Reports, and external review is drawn from the company’s own announcement. It is not a claim that every obligation in the revised policy became nonbinding.
- Zhu, “Silent Revision,” sec. 5 and appendix F. These examples are attributed to the paper’s document comparisons; this review did not independently reconstruct every underlying policy version.
- California, Senate Bill 53, Chapter 138, Statutes of 2025, approved September 29, 2025, Business and Professions Code secs. 22757.11(c), 22757.12(a)–(b), and 22757.15, https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=202520260SB53. Section 22757.12(a) expressly requires compliance with the framework; sec. 22757.15 provides for attorney-general enforcement and penalties up to $1 million per violation. The revision provision expressly requires a modified framework and justification, not an itemized directional inventory.
- Dylan Kresak, “Anthropic Floats Bill That Would Let Feds Step In Only When More Than 50 Lives Are On The Line,” Daily Caller, October 19, 2026, https://dailycaller.com/2026/10/19/anthropic-artificial-intelligence-governance-incident-safety-bill-aegis-act-anthony-cimino/. The report quotes the September 29 draft and reproduces Cimino’s October 18 statement. The legislative description here is based on that reporting, not an independently obtained complete draft or confirmation of subsequent revisions.
- Owen Dahlkamp, “Why Anthropic’s Draft Bill Sets $1 Billion Threshold for AI Risk,” POLITICO, October 21, 2026, https://www.politico.com/news/2026/10/21/anthropic-bill-ai-risk-threshold-01128458. The report distinguishes California’s assessment/disclosure use of its definition from the proposed federal model-blocking condition. Its body specifies more than 50 people and more than $1 billion; those formulations are used here rather than its less precise opening shorthand.
- Zhu, “Silent Revision,” sec. 6. The archive, directional inventory, replacement-protection explanation, and combined redline/plain-language format in the essay are the author’s recommended implementation of auditable revision, rather than a claim that the paper or current law mandates every element.