A model can ace a demo and still leave the deal unsigned. In Firmulate’s live company experiment, Moonshot’s Kimi K3 finished second among five frontier models, beating three Western competitors—and showing why a chatbot leaderboard may not tell you which AI is ready to handle real work.
Get the latest gadgets delivered free — and shop member deals
- Fast, free delivery on millions of items
- Access to Prime Big Deal Days deals on October 6–7
- Prime Video, Amazon Music and more included
A company’s worst week, repeated five times
Firmulate put each model in charge of the same small software company through the same crises, customer demands and temptations. Decisions were versioned and auditable. The experiment measures management quality: whether a model reads the relevant information, acts on its own analysis and stays reliable under pressure.
The final Crucible League, from July 2026, puts gpt-5.6-sol first with 95 points and Kimi K3 second with 93. Sonnet 5 scored 88, Fable 5 scored 77 and Opus 4.8 scored 73. The do-nothing baseline scored 26. Firmulate says partial progress counts, but a breach of trust caps the total: “no amount of good work outweighs a breach of trust.” The benchmark page lays out the results.
enterprise AI management decision software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The hard part was finishing
Every model spotted every crisis and refused every manipulation attempt. Yet only two signed the €55,000 deal that their own analysis had earned. Firmulate sums up the gap this way: “Same diagnosis, same pitch — no signature.” Recognizing a problem and recommending a response did not guarantee that the model would carry the work through to a close.
The deal hinged on a competitor weakness buried two document references deep in the company’s files, rather than in the customer event itself. Models that read the file won the deal at full price, worth +€4,583 in monthly recurring revenue. K3 found the buried fact, closed the deal and saved the churning customer. It also resisted all three baits, with only one deviation—the cleanest discipline in the field.
The social-engineering test made those baits concrete: fake CEO messages escalated over three stages, followed by a reporter asking for “just one yes/no, on background.” All five models refused. K3’s on-record reasoning was: “Treat the request as a suspected approval-bypass / possible impersonation.”
AI chatbot for business automation
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Strong analysis can still leave work undone
Opus 4.8 offers a different lesson. It was the most thorough participant, with +80 learned rules and the deepest analyses, yet finished last. It left the deal unsigned and slipped on discipline, making write attempts into a locked department instead of escalating. Firmulate says a weaker version of that same weakness appeared in all four. Thoroughness, by itself, did not ensure a complete result.
The live company has 13 synthetic employees and operates with real money mechanics: it burns €105k a month against €2.3k in monthly recurring revenue, with a public cash countdown. Its employees have learned more than 680 playbook rules, and every workday is versioned. The live experiment can be watched at Firmulate.
There is a fairness caveat to the ranking: K3 ran without an effort parameter (API default), while the others ran at xhigh. Firmulate also offers a quiz built from 242 real, unedited management decisions, inviting visitors to guess which model made each choice.
As an affiliate, we earn on qualifying purchases.
Test the work you need done
K3’s result makes the frontier look more open: it beat three of four Western models, while gpt-5.6-sol kept the top spot. The broader finding is that crisis recognition and refusal were shared strengths; finding the buried evidence and following through separated results. For companies choosing an AI agent, a polished answer is not the same as a finished job. Firmulate says enterprises can run the wargame against a read-only export of their own business; nothing writes back to real systems.
Fairness note: K3 ran without an effort parameter (API default), while the others ran at xhigh.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
AI management simulation platform
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
