Anthropic says its automated jailbreak tester beat human entries in 23 of 24 tests

Anthropic says its automated jailbreak tester beat human entries in 23 of 24 tests

Anthropic says an automated system matched or beat the best human submissions in 24 comparisons designed to test biological-weapons safety classifiers. It won 23 of those comparisons and tied the remaining one.

Diagram of the Constitutional Classifiers workflow

Anthropic’s earlier Constitutional Classifiers diagram explains the type of safeguard being tested; it is not a chart of the new results. Image: Anthropic

The result comes from an October 19 update to Anthropic's safety roadmap. The company says the automated runs took 24 to 48 hours, compared with weeks or months of time available to participants in the relevant bug bounties.

The comparison covers eight questions and three generations of classifiers

Anthropic evaluated the system against bounty work conducted between April 20 and September 11, 2026. The test used the same eight questions across three generations of production classifiers, producing 24 question-and-classifier combinations.

These classifiers are safeguards intended to identify dangerous requests. The experiment tested ways of getting past them; it was not a general contest measuring all cybersecurity work or every aspect of human safety research.

The company judged its automated findings more substantial in aggregate than the bounty findings. It therefore marked its internal red-teaming goal complete, ahead of a January 1, 2027 target.

Anthropic diagram showing three layers of agent safeguards: in the model, around the model, and last resort

Anthropic’s broader agent-safeguard diagram, shown as background to its safety research. Image: Anthropic

The published result is narrower than a general claim about AI safety

The comparison is Anthropic's own assessment. It has not yet published the more detailed follow-up it says it plans to release, so the short roadmap update does not provide a full independent evaluation of the method.

Matching the strongest submissions on these questions also does not show that every possible bypass has been found. It shows that automation performed well within the specific comparison Anthropic chose.

The work sits alongside other investigations into misuse. MIXED recently covered Anthropic's findings about harmful campaigns assembled from individually ordinary requests. That is a different failure mode from bypassing a biological-risk classifier.

For now, the concrete finding is a faster automated search that Anthropic says outperformed its bounty benchmark in nearly every tested pairing, with more methodological detail still to come.

Leave a Comment

Your email address will not be published. Required fields are marked *