OpenAI's GPT-6 Astra scored 55% on 11 Ironclad tasks, and the time savings are simulated

OpenAI's GPT-6 Astra scored 55% on 11 Ironclad tasks, and the time savings are simulated

OpenAI says GPT-6 Astra got 55.0% of the scoring criteria right across 11 contracting tasks built with the legal software firm Ironclad, against 41.6% for GPT-5.6 Sol. The same research post reports the work taking 19.2 minutes per attempt rather than 37.0, but a footnote on that table says the minutes are "simulated estimates based on assumed model processing and generation speeds, not measured customer time savings".

The Ironclad wordmark and green bracket logo on a dark background with folded abstract shapes

Ironclad's own brand card, a green bracket logo and wordmark over dark folded shapes. Image: Ironclad

The post went up on October 6 and was read at 23:11 UTC the same day. Both scores are OpenAI's own measurements of OpenAI's own models, taken with Max reasoning for Astra and High reasoning for Sol, which OpenAI describes as "the settings where each model scored highest".

Eleven tasks, scored against up to 50 criteria each

Ironclad employees and OpenAI staff who use Ironclad picked the tasks, which cover legal, commercial and procurement work. OpenAI names three of them: setting up nondisclosure agreements, creating procurement approval processes, and updating a reusable legal clause so it reflects the jurisdiction a requester selects.

Each task was graded against 8 to 50 criteria depending on complexity, which is where the 55.0% and 41.6% averages come from. OpenAI estimates an experienced user would need about 30 to 40 minutes per task, an estimate rather than a measured baseline.

Ironclad supplied hosted environments running its own product for the models to work in. OpenAI's researchers built synthetic training tasks from contracts in the SEC's EDGAR database, filtered to strip personal information, then used reinforcement learning to train against them. OpenAI states that no customer data, internal OpenAI contracts or nonpublic Ironclad contracts were used.

The minutes never came off a clock

The time figures are the softest numbers in the post, and OpenAI labels them as such. Nothing in the published figures records how long a person or a model actually took on a live Ironclad account, so the drop from 37.0 to 19.2 minutes describes a model of processing speed rather than an observed saving.

Two demonstration clips muddy this further. In the clip OpenAI published, Astra met about 94% of one task's criteria in an estimated 20 minutes and Sol about 85% in an estimated 32 minutes. Those are single-task figures; the averages across all 11 tasks are the 55.0% and 41.6% in the table.

An unnamed internal model scored 63.7%

OpenAI also reports, in the same post, that "an internal model used in the development of Astra achieved an even stronger 63.7% on these tasks". It does not name that model, has not released it, and says only that it aims "to bring these further gains to future models".

OpenAI's own reading of the results is cautious. Losing track of a business rule halfway through a task "limits what a software company can confidently ask it to do", the post says, and "this underscores why human oversight still matters as agents get better at complex contracting tasks".

Ironclad's chief technology officer, Sunita Verma, is quoted saying that "for AI to be genuinely useful in complex areas of contracting, it has to do more than complete individual actions", and that agents "need to understand the full contracting lifecycle". OpenAI is now inviting a small number of other software companies into the same arrangement, asking each for a concrete task its agents fail at and a definition of what success would look like.

The underlying capability has been shipping elsewhere in OpenAI's stack, including the hosted browser added to the Agents API on October 2, and OpenAI has already sold Astra into legal work through its invitation-only Astra for Law service. The Ironclad figures are the first time it has put a rubric score on that kind of work.

Leave a Comment

Your email address will not be published. Required fields are marked *