🔍 Read the full analysis: 71 AI Configurations Put My News Pipeline Choices To The Test on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Thorsten Meyer says he tested 71 AI configurations on English and German news articles between Sept. 8 and Oct. 13, 2026. GPT-6 Astra scored highest, but GPT-6 Luna was selected for production because it cost far less; two fallback models were removed after producing fabricated quotes or unusable output.
Thorsten Meyer says he tested 71 AI model configurations on real news sources and chose GPT-6 Luna for his production news pipeline, despite GPT-6 Astra earning the highest writing scores. In results published Oct. 13, Meyer reported that Astra cost about $0.12 per article, compared with roughly $0.0012 for Luna, while two backup models generated fabricated quotes or failed to produce usable articles.
The test ran from Sept. 8 to Oct. 13, 2026, using the production prompt Meyer says his pipeline uses. It covered API, subscription, free-tier and locally run open-weight models. Each configuration wrote English and German editions from two sources: a 6,340-character product filing and a 619-character political news item. Meyer said the shorter source better resembled much of his day-to-day work.
Two AI judges, GPT-6.1 Sol and Claude Opus 5.5, assessed the articles for source fidelity, fabrication, clarity about uncertainty, news value, structure and polish; German articles were also assessed for language quality. The strongest configurations then completed a further run of 10 articles, mostly based on short sources. Meyer reported a robustness score of 3.93 for Astra across 12 articles and 3.67 for Luna, while noting that rankings based on only two initial articles could be noisy.
Meyer said Astra scored 4.00 out of 5 in both languages in the standard test and received perfect source-fidelity scores there. Luna scored 3.67 in English and 3.68 in German in that test. In the larger run, Luna’s source fidelity was 4.71 out of 5, and every Luna article passed the pipeline’s output-validity check. Meyer reported a production median of about 17 seconds per Luna article, against roughly 50 seconds for Astra.
Infographic 01 · Who writes the best news article?
The best writer is not the one in production
71 model configurations, one real job: turn a news source into an article for an English and a German edition. GPT-6 Astra writes the best articles. GPT-6 Luna writes them for about a hundredth of the price.
Robustness run: 12 real news sources, English edition
Dot = mean score (1–5), bar = 95% bootstrap interval. German edition mean below each score. Overlapping bars are not a real difference.
Two judges, two caveats: GPT-6.1 Sol and Claude Opus 5.5 grade every article and were also tested as writers, so they partly grade themselves. Claude models ran with effort medium unless noted.
Source: Thorsten Meyer AI model-scout benchmark, 8 Sep – 13 Oct 2026. Task: write a news article from a real source with the production prompt, English and German editions. Judges: GPT-6.1 Sol (reasoning high) and Claude Opus 5.5, both with the source in context; score 1–5, mean of both judges. Robustness run: 2 test articles plus 10 more real sources, mostly short like production; mean with 95% bootstrap interval.
Powered by Thorsten Meyer AI
Cost and Accuracy Shape the Choice
The results illustrate a practical trade-off for publishers using AI to produce articles at scale: the top score may not justify the added expense if a less costly system meets the required quality bar. Meyer said Astra’s article cost was about 100 times Luna’s, while Luna’s output was consistently usable in his test. That makes his decision a pipeline-specific operational choice, not evidence that Luna is the best writer for every news organization.
The reported failures in backup systems carry a separate lesson. Meyer said GPT-4.1 nano produced invented quotations and received a source-fidelity score of 1.00 out of 5. He said locally run gpt-oss-120b produced zero valid articles out of four, with broken quotation formatting and invented named quotes. A fallback can therefore affect published material even when it is not the primary model; the article does not say whether any of these test outputs reached readers.
As an affiliate, we earn on qualifying purchases.
A Pipeline-Specific Model Test
Meyer framed the exercise as a test of task performance rather than a general model ranking. Public benchmarks can compare capabilities, he wrote, but do not by themselves establish which model best handles a particular publication’s prompts, source material and output requirements. His experiment used two source texts and two language editions, with the pipeline’s usual repair step included in the validity criterion.
The scoring design also included 95% bootstrap intervals for results, according to Meyer. He cautioned that, with only two articles in the initial test, score gaps below about 0.3 points should be treated as noise and described the larger robustness run as the more reliable ranking. These results therefore speak to Meyer’s workflow and sample, rather than establishing a general ranking across newsrooms or subjects.
The test also compared different reasoning-effort settings for Luna. Meyer reported that the highest setting took 211 seconds and cost about 1.43 cents per article, versus 17 seconds and 0.12 cents at the production setting, with no clear quality improvement. He said the score differences fell within the test’s uncertainty.
“The best model is not the one I run.”
— Thorsten Meyer
As an affiliate, we earn on qualifying purchases.
Limits of the Reported Results
The results come from Meyer’s own evaluation; the supplied account does not include independently verified testing or enough detail to reproduce every score. The test used two initial sources, with a further 10-article robustness run for the strongest configurations. Results may differ with other prompts, subjects, languages, model versions, judges or publication standards.
Meyer reported that score intervals for Astra and Luna did not overlap in the robustness run, but the supplied material does not provide full interval data for every model or a detailed breakdown of judge agreement. It also does not specify how often the pipeline uses each fallback, whether any fabricated quotes appeared in published articles, or the full process for human review before publication. The reported findings should not be read as proof that either model will perform the same way in other settings.
AI language model API subscription
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Luna Remains in Production
Meyer said the two low-performing fallback models were removed from the news route on Oct. 7 and that Luna was selected for the production workflow. The source account ends with an incomplete description of the revised failover arrangement, so it does not establish what backup now follows Luna or how the system handles a failure.
The next practical test is whether Luna continues to meet Meyer’s quality and validity standards on a broader stream of production sources. The report does not provide a planned date for another evaluation, and it does not say whether the publisher intends to add further models or publish ongoing results. Readers should treat the reported costs and scores as measurements from this test period, not fixed guarantees for future model versions or usage.
As an affiliate, we earn on qualifying purchases.
Key Questions
Which model scored highest in Meyer’s test?
GPT-6 Astra scored 4.00 out of 5 in both English and German in the standard test. Meyer reported a score of 3.93 across 12 articles in the robustness run.
Why did Meyer choose GPT-6 Luna for production?
Meyer said Luna produced consistently usable articles at about $0.0012 per article, compared with roughly $0.12 for Astra. His choice reflects the costs and quality thresholds of his own pipeline.
What did the test find about reasoning effort?
Meyer tested six Luna reasoning-effort settings and reported no clear quality gain from the more expensive settings. The maximum setting cost and took about 12 times as much as the production setting, according to his measurements.
What went wrong with the fallback models?
Meyer said GPT-4.1 nano invented quotations, while locally run gpt-oss-120b produced no valid articles in four attempts and also generated invented named quotes. He said both were removed from the news route on Oct. 7.
Do these results establish which model is best for all newsrooms?
No. The findings describe one publisher’s test, using a particular prompt, two initial sources, English and German editions, and selected follow-up articles. Other workflows and source material may produce different results.
Source: ThorstenMeyerAI.com
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
