🔍 Read the full analysis: Zero Legal Score: The Result Of Our Home AI Weight Editing Test on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A home experiment edited the weights of the local AI model Ornith 1.5 35B-A3B to reduce looping. On 10 published legal benchmark tasks, the edited model completed all 10, compared with 4 on the same setup before editing, but its legal score remained zero. The test was limited, and the unchanged model completed 9 of 10 when run with its recommended sampling settings.
A home test of Ornith 1.5 35B-A3B found that editing the model’s weights helped it finish legal benchmark tasks but did not improve its legal score: on the 10 published Lexassay tasks, the edited model answered 10 of 10 on the test setup, while scoring zero. The result suggests that reducing repetitive output can address a model’s tendency to loop without supplying the legal knowledge needed for accurate answers.
The experiment was conducted from Wednesday evening to Thursday morning and was inspired by MIT researchers’ work on what they call “weight-space programming.” It tested a related but simpler technique, not a replication of the MIT method. The test used Claude Opus 5.5 to write code that ran the model, measured a direction associated with looping, and removed it at half strength from the output weights of 27 of the model’s 40 layers. The researchers say there was no additional training or legal data, and no AI read the weights as text.
On the same setup, the edited model completed 10 of 10 tasks, up from 4 of 10 before the edit. But among the edited answers, the experiment’s grading found only 7 of 50 checked criteria were met, compared with 3 before editing, and recorded 55 wrong or invented citations. The published article reports a legal score of zero both before and after the edit. These figures refer to the 10 published tasks; five other Lexassay tasks remain hidden.
The report also warns that the comparison is affected by sampling settings. With the model-card-recommended settings and no weight edit, Ornith answered 9 of 10 tasks on its original setup. The report says that setup and the edited-model setup are not directly comparable; on the original published run, the unchanged model answered 8 of 10. The experiment’s authors disclose that Claude carried out the edit and Claude agents performed blind grading. OpenAI’s GPT-6 Astra, running in Codex, separately reviewed the code, results and claims and identified faults.
Infographic 01 · What we changed
One loop direction, weakened at half strength
Inspired by the MIT post, but a simpler, classic technique (directional ablation): code written by a Claude Opus 5.5 session measured one loop direction by running the model and projected it out of the weights. No training, and no AI reading weights as text.
The six steps
- Write test material that is not the benchmark: 24 legal development questions plus 8 general controls, 0 overlap with benchmark prompts
- Find the true loop onset at token level: from there, at least 90% of the next 1,500 positions repeat earlier 32-token sequences
- Run the model and measure one loop direction per layer: average internal signal inside the loop minus the reasoning just before it
- Weaken it in the weights:
W ← W − α·r·(rᵀW)on the MLP output matrices of all 256 experts and the shared expert - Bake an edited copy; the original files stay untouched and the edited weights stay internal
- Benchmark original and edit on the same local setup, graded blind by three Claude agents; figures cover the 10 published tasks
Quick test: share of pure copying
Restart 20 tokens before a known loop onset, 8 cases, 3,000 tokens. Lower is better.
The cost of the edit
- Full strength: 23 of 24 legal questions finished, but control answers broke: tokens dropped out of short formulas, and month counting went wrong
- Half strength: 23 of 24 finished, 9 of 9 checked control facts right, and one German control question was answered in Chinese once
- Before the edit: 8 of 24 legal development questions did not finish; all 8 controls did
One run per setting. The loop detector was revised three times during the experiment, so the development numbers come from a moving yardstick. The direction is a statistical description, not a single “loop neuron”.
Sources: our experiment records, 4–5 Nov 2026 (executing Claude Opus 5.5 session; review by GPT-6 Astra in Codex); Ornith-1.5-35B-A3B model card.
Powered by Thorsten Meyer AI
Fewer Loops, Not Better Legal Answers
The result separates two problems that can look similar in a benchmark: a model may fail to answer because it gets stuck generating repetitive text, or because it lacks the knowledge to answer correctly. The edit appears to have helped with the first problem: the model completed more tasks on the experiment’s setup. The unchanged legal score and the reported citation errors show that completion is not the same as correctness.
That distinction matters to people evaluating local AI for legal research. A system that produces a complete response may still cite nonexistent or incorrect legal authorities. Lexassay checks citations against primary sources, and its reported results show that the edited model’s fluency did not translate into dependable legal performance. The test is an experiment, not legal advice, and does not establish how the method would perform on other models or tasks.
The comparison with recommended sampling settings also cautions against attributing every improvement to a weight change. In this case, changing the model’s sampling configuration without editing its weights was associated with more completed tasks than the original setup. The report does not establish that the edit was responsible for all of the observed increase.
Products Worth Considering
Samsung 37-inch Odyssey G7 4K UHD Curved Gaming Monitor
Apple AirPods Pro 3 Wireless Earbuds with Active Noise Cancellation
Logitech MX Master 4 Ergonomic Wireless Mouse with Haptic Feedback, Graphite
As an affiliate, we earn on qualifying purchases.
The MIT Work Behind the Test
The test followed a report by four MIT CSAIL researchers: Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab. Their work described an approach in which one general-purpose language model examines another model’s weights and edits them directly. In a highlighted Qwen case, the MIT team reported that a model’s looping was linked to a direction in its internal signals; after an edit, its mathematics score rose from 7.9% to 48.1%. The home experiment drew on the broader idea that an edit might remove an obstacle to an existing ability, rather than add a missing ability.
The home test used directional ablation, a technique that measures a signal by running a model and then removes a direction from its weights. The report says this differs from the MIT team’s weight-space approach, where GPT-6 Astra reasoned about the weights directly. The authors chose Ornith because, among four local models considered, it showed the relevant looping behavior and could be run on the available machine. Its publisher describes it as a coding- and agent-focused mixture-of-experts model, not a model built for legal work.
Lexassay asks models the same questions on German, US and EU law without internet access or tools, and checks cited statutes and court decisions against primary sources. The benchmark has 15 questions in total: 10 published tasks used for these reported figures and five hidden tasks intended to prevent tuning against every question. The article says the runs, answers, grades and citation checks are public on Lexassay.
“The edit repairs finishing, not legal knowledge. “The model finishes” is not “the answer is right”.”
— Thorsten Meyer, author of the home experiment
Limits of the Ten-Task Results
The experiment reports one run per setting, a small development set and a loop detector that changed during the work. Those limits make it difficult to tell how repeatable the results are or how much of the task-completion difference came from the edit rather than other setup choices. The original and edited runs used two local setups that the report says are not directly comparable.
The grading also has a disclosed conflict: Claude conducted the edit, and Claude agents from the same model family performed the blind grading. GPT-6 Astra reviewed the work and found real faults, but this was one independent reviewer, not a second experiment. The report does not provide evidence that the edited model would score better on the five hidden Lexassay tasks or on legal questions outside this benchmark.
The precise cause of the model’s poor legal performance is not fully isolated by this test. The authors interpret the unchanged zero score and citation problems as evidence that the edit reduced looping rather than supplied legal knowledge. That is consistent with the results reported, but a single limited experiment cannot establish how the technique will affect other models or settings.
Public Runs and Further Testing
The experiment’s runs are available on Lexassay.com, including the original run, the model-card sampling run, the weight-edited run and an edit control. The report also points readers to a method page describing the procedure. These records let readers inspect the model’s answers, grading and citation checks rather than relying only on the summary figures.
The next useful test would repeat the comparison across more runs while keeping the model, hardware, token limits and sampling settings consistent. Independent grading and evaluation on the five hidden tasks would help show whether the improvement in completion is reproducible and whether it changes performance on questions not used in the published set. Until then, the reported finding is limited: the edit reduced a looping problem in this test, while the model’s legal score remained zero.
Key Questions
What did the weight edit change?
The experiment measured a direction associated with looping and removed it at half strength from the output weights of 27 of Ornith’s 40 layers. The report describes this as directional ablation, not the direct weight-reading method used in the MIT work.
Did the edit improve the model’s legal accuracy?
Not according to the reported overall score. The legal score remained zero. The edited model met 7 of 50 checked criteria, up from 3, and its answers included 55 wrong or invented citations.
How many benchmark tasks did the edited model answer?
It answered 10 of the 10 published tasks on the edited test setup, compared with 4 of 10 on the same setup before editing. The benchmark also has five hidden tasks, which are not included in those figures.
Could sampling settings explain some of the improvement?
They may affect the comparison. Without a weight edit, the model answered 9 of 10 tasks using the settings recommended in its model card. The report says the two local setups were not directly comparable, so the figures do not isolate the effect of the edit perfectly.
Was this a repeatable or independently replicated test?
The report describes one run per setting, not a replication. OpenAI’s GPT-6 Astra reviewed the code and claims and found faults, but it was a review of this experiment rather than a separate competing test.
Source: ThorstenMeyerAI.com
