🔍 Read the full analysis: The AI For Law Scored 8/100 When We Asked 15 Legal Questions on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A legal AI model from Z.ai scored 8 out of 100 on Lexassay, a test of 15 legal questions across German, U.S. and EU law. The test operator reports citation errors and says the results are a spot check, not a definitive ranking; the operator also disclosed that Claude models grade the answers and that a Claude model leads the published results.
GLM-5.3 FlashX Legal scored 8 out of 100 on a 15-question legal AI test run by Thorsten Meyer AI, which says the model fully solved none of the tasks and gave no answer to two. The result is a limited test of models answering without internet access or tools, not a broad measure of how legal AI systems perform in real workflows.
The test, called Lexassay, covers German, U.S. and EU law. Its questions include calculating court deadlines, applying recent legal changes, identifying leading court decisions and spotting false premises. Ten questions are public and five are hidden; for the hidden questions, the operator publishes scores but not answers. In all, the source says 42 AI setups were given the same questions.
Thorsten Meyer AI reports that GLM-5.3 FlashX Legal had 21 citations judged real, 25 judged misattributed and one judged invented. Under Lexassay’s scoring rules, a real legal reference can still lose credit if it does not support the model’s claim. The operator says two answers received no response after the model used its full 64,000-token thinking budget. Its base model scored 3 out of 100, according to the same test results.
The report also gives scores for other systems. Mistral Large 4 scored 10, while Claude Opus 5.5 led the published results with 79, followed by GPT-6 Astra at 72 and GPT-6.1 Sol at 71. Even the leading model fully met every criterion on only eight of the 10 public tasks, the report says. These figures are the test operator’s findings, not independently verified results.
Infographic 01 · The leaderboard
Same 15 legal questions, very different results
German, US and EU law, no internet, no tools. Every cited law and ruling checked against the source.
The top · 67 to 79
The middle · 41 to 59
The weak group · 0 to 35
Selection of 14 of 42 configurations. Score 0–100 = share of achievable points, with deductions for wrong citations. Small local models (1–12) not shown.
A small spot check: one answer per model and task, no tools, no web. General information, not legal advice. Disclosure: Thorsten Meyer AI runs Lexassay; Claude agents grade blind.
Source: Lexassay, release of 4 Nov 2026.
Powered by Thorsten Meyer AI
Why Legal Citation Errors Matter
The results highlight a practical risk for people who use chatbots to understand legal rights or act on a deadline: an answer can sound plausible while citing a decision or law that does not support the stated conclusion. Lexassay’s rules penalize invented and misattributed citations, but a score from this test cannot establish how often errors occur across all legal questions or in systems that can search authoritative sources.
The report illustrates the stakes with court deadlines. It describes two German civil cases with judgments served on April 27, 2027, one in Munich and one in Berlin. The source says a public holiday in Bavaria moves the Munich appeal deadline to May 28, while the Berlin deadline remains May 27; the written appeal grounds are due June 28 in both cases. The example is part of the test material, not evidence that a person should rely on a chatbot for a live case. Missing a legal deadline can affect a case, so the report advises checking law and court rulings directly and contacting a lawyer when a deadline, money or an authority is involved.
Products Worth Considering
Apple MacBook Air 13.6-inch with M5 Chip, 16GB Memory, 512GB SSD, Sky Blue
Bose SoundLink Flex Bluetooth Speaker (2nd Gen) – Portable Outdoor Speaker with Hi‑Fi Audio, Waterproof and Dustproof, USB‑C, Up to 12 Hours
Logitech MX Master 4 Ergonomic Wireless Mouse with Haptic Feedback, Graphite
As an affiliate, we earn on qualifying purchases.
How Lexassay Scores Legal Answers
Lexassay compares each response with a prepared answer containing three to six weighted criteria. The test operator says every citation is checked against official databases and classified as real, misattributed, invented or not findable. A citation that exists but does not say what the model claims is treated differently from a fabricated reference. Answers also lose five percent if they lack a legal-advice disclaimer or a pointer to a lawyer, according to the method description.
The models answer without web access or other tools, and different thinking-effort settings are tested separately. Meyer says the design measures what a model can produce on its own, roughly like a chatbot used without search. The report explicitly calls the exercise a spot check rather than a strict benchmark: each model receives one answer attempt per task, so small score differences may reflect noise. Results from systems that can consult supplied documents or online sources are not directly comparable.
The source contrasts its findings with Vals AI’s legal-agent benchmark, where models work with supplied documents. It says Mistral Large 4 ranked eighth there, ahead of Claude Opus 5.5 at 21st and GPT-6 Astra at 24th. That different ordering underscores that benchmark outcomes depend on the task and setup; it does not establish that one model is generally better for legal work.
““Optimized for professional legal workflows, with a focus on document understanding, legal reasoning, and task execution.””
— Z.ai, in developer documentation quoted by Thorsten Meyer AI
Limits of the Published Results
Thorsten Meyer AI runs Lexassay, creating a disclosed conflict of interest. The source says several Claude agents graded answers and a Claude model ranked first. It also says the test method and public answers, gold answers and citation checks are published for scrutiny. Those disclosures allow readers to examine the process, but do not amount to an independent replication.
The source does not provide enough information here to establish how representative 15 questions are of legal work, how scores would change across repeated runs, or how these models compare when connected to legal databases and document tools. The five hidden questions also have scores published without their answers, limiting outside review of that portion. The report says GLM-5.3 FlashX Legal ranked seventh of 81 on another measure, but the supplied material cuts off before explaining that ranking, so its basis and comparability are unclear.
The test’s examples and scoring reflect the operator’s methodology. The findings do not establish that the model is unsafe or unusable in every legal setting, nor do they show how a lawyer or client would perform with the system as part of a supervised workflow. The article is not legal advice, and the reported scores should not be treated as a substitute for case-specific guidance.
What Readers Should Check Next
Readers considering AI for legal work should check whether a system can retrieve current primary sources, then verify each cited law or ruling against the original official text. Lexassay’s operator recommends using AI as a starting point rather than an authority, and not taking a filing deadline from a chatbot. Anyone facing an active deadline, financial claim or communication with a government body should seek advice from a qualified lawyer.
Further comparisons would be more informative if they used repeated runs, clearly published scoring rules and independent grading, and tested both tool-free chatbots and systems with access to legal documents. The supplied report does not announce a date for another Lexassay release or say whether Z.ai plans to respond to the score. For now, the result is a warning about performance on this particular set of questions, not a final verdict on legal AI.
Key Questions
What did GLM-5.3 FlashX Legal score?
Thorsten Meyer AI reports a score of 8 out of 100 on its 15-question Lexassay test. The operator says the model fully solved none of the tasks and gave no answer to two.
What did the test measure?
Lexassay tested answers to legal questions across German, U.S. and EU law, including deadline calculations, legal changes, court decisions and false premises. Models answered without internet access or tools.
Can the results show which legal AI is best?
No. The operator describes Lexassay as a spot check with one answer per model and task. Scores may be affected by the limited question set, and results can differ when models use documents or search tools.
Why does the report disclose a conflict of interest?
Thorsten Meyer AI runs Lexassay, and the report says Claude agents graded the answers while a Claude model ranked first. The operator says it publishes its method and test materials, but the disclosed relationship is relevant when evaluating the findings.
Should I use a chatbot to calculate a legal deadline?
The report advises readers not to rely on a chatbot for a deadline. Check the relevant law and court information, and consult a lawyer about an active case or filing date.
Source: ThorstenMeyerAI.com
