🔍 Read the full analysis: Claude’s Model Edit Under Review: What GPT-6 Astra Caught on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A GPT-6 Astra session reviewed a Claude Opus 5.5 experiment that edited the weights of the local model Ornith 1.5. The reviewer reproduced problems with resuming runs, task-ID validation, command-line parsing and weight conversion; after fixes, all 12 author tests passed, while four of six reviewer provenance checks still failed.
A review by OpenAI’s GPT-6 Astra found reproducible faults in a Claude Opus 5.5-led experiment that edited the weights of local model Ornith 1.5, including a resume bug that could reuse old results and a conversion that changed weights even when the edit strength was zero. The review did not run a competing model-improvement experiment; it checked the implementation and evidence behind a reported benchmark result.
The experiment used a directional ablation inspired by an MIT CSAIL post on weight-space programming. According to ThorstenMeyerAI.com, the Claude session identified a “loop direction” in Ornith 1.5 and weakened it in the model weights without training. On the 10 published Lexassay tasks, the edited model answered all 10, compared with four for the unedited model. Its legal score remained zero. The task runs are public on Lexassay, the source says.
GPT-6 Astra, running in Codex, was assigned to review and validate the other session’s implementation. It used separate check programs, left the main implementation untouched, and did not inspect hidden benchmark prompts, topics, answers or reference solutions, according to its report as summarized by the article. The Claude session made the resulting changes. The review found that resumed runs could treat an earlier answer as complete without checking whether the model or settings had changed; unknown task IDs could produce a successful run with no evaluations; and a comma conflict broke the actual command line even though parser unit tests passed.
The reviewer also found that converting weights back to a compact format changed values at zero edit strength. In its first test, it reported differences in 105,315 of 1,048,576 reconstructed values, or 0.84% of the matrix’s overall size. The article says the team changed the command syntax, made unknown IDs fail, and stored edited tensors in higher-precision bf16 rather than re-quantising them. In a later measurement, converting one matrix to a different number format moved it by 0.51% and changed 915,360 of 1,048,576 values. Those figures describe conversion behavior, not legal performance or a general model error rate.
Infographic 01 · The two review rounds
Green tests, real faults
GPT-6 Astra (OpenAI, in Codex) reviewed a Claude Opus 5.5 session's model-edit experiment while it was still being built. What it found, and what happened next.
Round 1 · 4 Nov · four faults
- Resuming a run could replace a different experimentAnswers were skipped even after model, settings or edit changed; in the reviewer’s test: 0 new inferences, old answer kept! Partly fixed
- A mistyped task ID gave a “successful” empty runZero evaluations, no error✓ Fixed
- Unit tests green, real command brokenCommas in the sampling syntax clashed with the model list; 3 parser tests passed, the real dry run failed✓ Fixed
- Conversion changed weights at edit strength zeroRe-quantising one real matrix: 0.84% change, 105,315 of 1,048,576 values differed! Partly fixed
Round 2 · 4 Nov, evening · the fixes re-checked
- Old result rows got a new historyAccepted after the requested seed was changed to 999✓ Solved another way
- Loop-detector settings not fingerprintedChanging them left the configuration “identical”✓ Solved another way
- Model configuration and chat template invisibleOnly file names, sizes and modification times were compared✓ Solved another way
- bf16 storage still changes the computationThe zero-strength test compared a conversion with itself→ Control added
✓ Fixed with a test that fails on the old code · ! Partly fixed in round 1, finished in round 2 · ✓ Solved another way: resuming was removed entirely, and every run records fingerprints of configuration, tokenizer and code · → Control added: a conversion-only copy with no edit. The reviewer changed none of the main code and read none of the hidden benchmark tasks.
Sources: our experiment records, 4–5 Nov 2026: reviews of 4 Nov and final explanation of 5 Nov by GPT-6 Astra in Codex (OpenAI); results report of the executing Claude Opus 5.5 session (Anthropic).
Powered by Thorsten Meyer AI
Why the Review Changed the Result’s Meaning
The review matters because benchmark results depend on more than the final score: the test rig must reliably run the intended tasks and preserve the distinction between an edited model and a conversion artifact. If a resumed run can silently reuse earlier answers, or a typo can produce an apparently successful empty run, a green status alone does not establish that the reported evaluation happened as described.
Here, the conversion check also bears on attribution. If weights change when edit strength is zero, some observed effect might come from changing the representation rather than from the intended edit. The source says the reviewer’s first test found such a change and that the implementation was altered afterward. That finding does not, by itself, show that the published benchmark outcome was caused by conversion; it identifies a confound that needed to be tested and reported.
The result remains narrow: a gain in the count of answered published tasks did not raise the legal score above zero. The case illustrates why readers should distinguish task completion from task quality, and why public runs, controls and provenance checks matter when evaluating claims about model editing.
Products Worth Considering
Samsung Galaxy Tab A11+ Kids Edition, 128 GB Android Tablet, Royal Blue
Jackery Explorer 2000 v2 Portable Power Station with Two 200W Solar Panels
Jackery Solar Generator 1000 v2 with 100W Solar Panel
As an affiliate, we earn on qualifying purchases.
How the Two AI Sessions Were Split
The work began on November 4, 2026, with two AI sessions involved. Once the participants recognized they were working on the same project, their roles were separated. Claude handled the experiment plan, test rig, model edit, benchmark runs and blind grading by three Claude agents. GPT-6 Astra was assigned a review role instead of producing a second edited model. The source says preparatory files from Astra’s earlier work were set aside after that decision.
The review was conducted in rounds while the experiment was being built. In its first round, Astra reported four reproducible faults, including the three execution problems and the zero-strength conversion issue. The author says the Claude session addressed each with a test designed to fail against the old code. The source article was updated on November 9 to clarify that its figures refer to the 10 published Lexassay tasks and that the runs are available publicly.
The later review was not a clean bill of health. The source reports that all 12 of the author’s tests passed in the second round, while four of the reviewer’s six provenance checks failed. It also says Astra corrected some descriptions in the write-up: “exactly identical at strength zero” was inaccurate, a per-answer average of wrong citations needed qualification, “parallel Codex experiment” overstated its role, and the causal claim drawn from one control was stronger than the evidence. These are the source author’s account of the review, not an independent audit of the project.
What the Review Does Not Establish
The source does not establish that the model edit improved legal reasoning: it reports a legal score of zero, despite the increase from four to 10 answered published tasks. Nor does the conversion measurement alone establish how much of the benchmark result, if any, was caused by the original conversion issue. The article says the implementation was changed, but does not provide enough detail here to independently quantify the effect of that fix on the final benchmark outcome.
The review also had limits. According to the source, GPT-6 Astra did not rerun every test during its final documentation step, did not act as a fourth blind grader, and took its measurements on a CPU rather than the production GPU path. The source says hidden benchmark material was not inspected, but the account provided does not include an independent audit of that process. The four failed provenance checks in the second round also remain a reported finding; the supplied material does not specify each check or say whether all were later resolved.
Public Runs and Follow-Up Checks
The immediate reference point is the public Lexassay material, which the source says includes the weight-edited runs, an edit control and model-card sampling. Readers can compare those records with the updated implementation and the 10 published tasks, while keeping the reported legal score separate from the task-answer count.
The source material does not announce a further experiment or a scheduled review. The next useful evidence would be a clearly documented account of the remaining provenance-check failures, along with tests showing that the corrected pipeline produces valid, traceable runs under the relevant execution path. Until then, the reported benchmark change should be read alongside the reviewer’s caveats rather than as proof of improved legal performance.
Key Questions
What did Claude’s experiment change?
According to ThorstenMeyerAI.com, Claude Opus 5.5 used a directional ablation to weaken a “loop direction” in Ornith 1.5’s weights, without training the model.
What did GPT-6 Astra find?
The review reproduced faults involving resumed runs, invalid task IDs, command-line parsing and weight conversion at zero edit strength. The source also reports four failed provenance checks in the second review round.
Did the edited model perform better at law?
The edited model answered 10 of the 10 published Lexassay tasks, compared with four for the unedited model, according to the source. Its reported legal score stayed at zero, so the task count should not be treated as evidence of improved legal performance.
Did GPT-6 Astra edit a second model?
No second edited model came out of Astra’s review role, according to the article. Astra checked and measured the implementation; the Claude session made the fixes.
Are the experiment results public?
The source says the 10 published task runs are available on Lexassay, along with the weight-edited run, an edit control and model-card sampling. The material provided does not settle every concern raised in the review.
Source: ThorstenMeyerAI.com
