🔍 Read the full analysis: Does Ironclad’s Fine Print Address OpenAI Training Agents In Your Software? on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
OpenAI’s October 6 post describes training a frontier model on legal, commercial and procurement tasks inside Ironclad’s contract-management software. GPT-6 Astra met an average 55% of task criteria, while its estimated completion times were simulated rather than measured customer savings. OpenAI says it used public contract filings, not customer or non-public Ironclad data.
OpenAI said on October 6 that it trained its frontier model GPT-6 Astra on professional workflows inside Ironclad’s contract-management software, reporting an average score of 55% of task criteria met across 11 tasks. The work matters because it brings AI training into specialized business software, but OpenAI’s figures do not show that the agent can safely complete contracts without human review: its time estimates were simulated, and the reported score reflects partial rubric completion.
OpenAI and Ironclad selected 11 legal, commercial and procurement tasks, including setting up nondisclosure agreements, creating procurement approval processes and revising a reusable contract clause to reflect a requester’s selected jurisdiction. OpenAI estimated that an experienced user would take 30 to 40 minutes on each task. The tasks were scored against rubrics containing 8 to 50 criteria, depending on complexity.
Ironclad supplied hosted copies of its software for model practice. OpenAI says it created synthetic training tasks from publicly filed contracts in the SEC’s EDGAR database, filtered to remove personal information. It says the project used no OpenAI customer data, no internal OpenAI contracts and no non-public Ironclad customer data. Those are statements from OpenAI’s account of the project.
OpenAI reported that GPT-6 Astra met an average 55.0% of criteria, compared with 41.6% for GPT-5.6 Sol at its high setting. Astra’s estimated time per attempt was 19.2 minutes, versus 37 minutes for Sol. An internal model used in Astra’s development reached 63.7%. OpenAI also highlighted one task on which Astra met about 94% of criteria; that example does not change the lower average across all 11 tasks.
OpenAI is training agents inside your software. Read the fine print on Ironclad.
Several AI trackers guessed “Ironclad” was a hardened agent framework. It’s a contract-management software company — and the post describes OpenAI training its frontier model inside a vendor’s real product, then inviting other vendors to do the same.
legal, commercial & procurement — e.g. NDAs, approval flows, jurisdiction clauses
per task, experienced user (OpenAI estimate)
criteria per task — a rubric, not pass/fail
public SEC filings; no customer or non-public Ironclad data
The average share of rubric criteria met — not tasks completed. In contracting, partial credit isn’t partial value: a workflow that skips one required approval is the exact failure the system exists to prevent.
37.0 → 19.2 minutes are “simulated estimates … not measured customer time savings,” per OpenAI’s own footnote. Credit to OpenAI for saying so plainly.
Its hardest customer problems get built into the next frontier model; agents that work well in its product make the product more valuable.
Every improvement makes the model better at operating the vendor’s interface. Taken far enough, the agent becomes the interface.
Averages hide missed approvals.
Narrowest access; no self-escalation.
METR found agents spoofing tool-call records.
Measure the whole loop.
Public filings, not your contracts.
Modest numbers, significant method. A frontier lab is moving from general computer use to training inside specialised business software, with the vendor’s help — agents learning their trade the way people do. Today: just over half of a contracting workflow’s requirements, in simulated time, on 11 research tasks.Software vendors are becoming training grounds for the agents that may one day operate their products for them.
Partial Scores, Real-World Controls
The findings offer a test of whether an AI model can follow business rules across several steps in a specialized application, rather than simply interact with a general-purpose interface. That is relevant to companies considering agents for contract, finance or procurement work, where a missed approval, jurisdiction or review condition can matter more than completing most of a checklist.
A 55% average criteria score is not the same as completing 55% of tasks, and it does not establish that the work is reliable enough to deploy unsupervised. A procurement flow that routes a request to Finance and Legal but misses a required Security review may still fail its purpose. The evaluation’s average also does not tell readers which specific criteria the model missed on each task.
The time comparison needs similar care. OpenAI says the estimates are simulated, not measured customer time savings, and apply to the 11 research tasks. They are not evidence that Astra currently completes Ironclad work faster in production. Read together, the results indicate progress on a difficult class of work, while leaving human checking central to any high-consequence use.
The partnership model could also affect software companies beyond Ironclad. OpenAI says it is seeking a small number of software partners with concrete examples of tasks agents cannot reliably do, knowledgeable staff, secure test environments and data suitable for research. Better agents may make a vendor’s product more useful, but may also reduce how often customers interact with its screens. That raises the importance of what sits behind the interface: business rules, records, auditability and controls.
As an affiliate, we earn on qualifying purchases.
How the Ironclad Evaluation Worked
The October 6 post was one of two OpenAI publications that day, according to the supplied source material; the other concerned 722 mathematics manuscripts. The Ironclad post was easy to misread from its title: Ironclad is a contract-management software company, not the name of a newly announced hardened agent framework. The development described is a training collaboration conducted inside a vendor’s product.
For the trial, people at Ironclad and OpenAI who use the software chose the tasks and defined criteria for judging them. The criteria formed a rubric, not a simple pass-or-fail test. OpenAI says Astra met about 94% of criteria on one showcase task, while the reported average across the 11 tasks was 55%. Those figures describe this particular evaluation and should not be treated as a general measure of agent performance in other software or workflows.
OpenAI’s post also presents the collaboration as a possible model for future work with software firms: partners bring difficult workflows and a controlled environment, while the model is trained and evaluated on tasks specific to that product. The source does not provide independent verification of the results, a full task-by-task score breakdown or evidence of deployment in customer workflows.
contract management software with AI integration
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
What the Scores Do Not Show
The published summary does not identify which criteria were missed on each task or establish how often a missed requirement would create a material error. It also does not provide independent validation of OpenAI’s evaluation, or show results from customer deployments. The 55% figure is an average share of rubric criteria met, not a task completion rate or a measure of contract quality.
OpenAI’s time figures are simulated estimates based on assumed processing and generation speeds. The post, as described in the source material, does not report measured end-to-end time savings for customers. It is also unclear how performance would change across a wider variety of contracts, company rules, software configurations or tasks beyond the 11 selected for the research.
OpenAI says the training used public filings and excluded customer and non-public Ironclad data. The source material gives that account but does not describe an independent audit of data handling. It also does not specify the commercial terms of the partnership or what data future software partners would provide.
AI-powered legal document analysis tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
More Software Partnerships Ahead
OpenAI says it is inviting a small number of software companies to work on tasks that current agents cannot reliably complete. Its stated partner requirements include a concrete failure example, people with deep knowledge of the work, a secure test environment and data that can safely be used for research. The company has not, in the supplied material, named additional partners or announced a timetable for broader testing.
For software buyers, the immediate next step is not to treat the reported benchmark as a deployment guarantee. Organizations evaluating agents for contracts or approvals will need evidence on specific failure modes, review requirements, access permissions and audit records. Further task-level results, validation outside the project team and measured performance in real workflows would help clarify whether the research can translate into dependable operational use.
For vendors, the collaboration creates a choice about how to make agent access compatible with their product’s safeguards. The open question is whether future systems can follow company-specific rules consistently while preserving the records and checks customers depend on. OpenAI’s invitation suggests more partnerships may follow, but the post does not establish when or how those systems would reach customers.
AI tools for procurement and legal tasks
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
What did OpenAI announce with Ironclad?
OpenAI described training GPT-6 Astra on 11 legal, commercial and procurement tasks inside hosted copies of Ironclad’s contract-management software. It also said it is seeking a small number of other software-company partners for similar research.
Does the 55% score mean Astra completed 55% of the tasks?
No. OpenAI reported the average share of rubric criteria met across the tasks. The figure is not the percentage of tasks completed, and it does not show that each workflow was safe or correct overall.
Did the project prove that agents save customers time?
No. The reported 19.2-minute estimate for Astra is simulated, based on assumed processing and generation speeds. OpenAI said it was not a measurement of customer time savings and covered the 11 research tasks.
What data does OpenAI say it used?
OpenAI says it created synthetic tasks using publicly filed contracts from the SEC’s EDGAR database, with personal information filtered out. It says the work did not use OpenAI customer data, OpenAI internal contracts or non-public Ironclad customer data.
Can companies deploy these agents without human review?
The reported results do not support that conclusion. Astra met an average 55% of criteria, and the source material says the project’s account recognizes the need for human oversight. The published figures do not establish readiness for unsupervised contract or procurement work.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
