OpenAI says people linked to Moonshot AI ran a campaign to extract its reasoning

OpenAI says people linked to Moonshot AI ran a campaign to extract its reasoning

OpenAI says it has disrupted a coordinated campaign to pull the protected reasoning out of its models, and it attributes "a core cluster of the activity to individuals associated with Moonshot AI, the developer of Kimi". The account, published on September 30, dates the earliest activity it saw to the first week of July and says the cluster was fully disrupted by July 28.

OpenAI chief executive Sam Altman in front of a large white OpenAI wordmark.

The figures OpenAI gives are attempts, not results. It says they "describe attempted, not necessarily successful, extractions", so nothing in its account shows how much hidden reasoning anyone actually walked away with.

The operators worked one conversation against another

Protected reasoning is what OpenAI calls "the model's internal record for working through a task", and it says extracting that record "can reveal information withheld from the final answer and help others reproduce the model's capabilities". The technique it describes was to copy encrypted reasoning out of one conversation, then ask a model in a second conversation to decrypt and transcribe it.

OpenAI is specific about what it says did not happen. The operators "did not break our encryption, compromise a database, or gain direct access to stored user conversations", it says, and instead "manipulated model interactions" so that protected reasoning surfaced in a form the requester could read, at a scale that broke its terms of service.

The volume spiked on two days in July

OpenAI dates the start to July 1, "initially at a low volume", and says the traffic then jumped: "high-volume spikes on July 24 and 25 consisting of 16,000 requests using a relevant extraction pattern from over 4,000 users". A wider sweep turned up related prompt-pattern activity across a cluster of more than 15,000 users, which it says it fully disrupted by July 28.

Outside help came into it too. OpenAI says independent security researchers reported related cross-model and conversation-compaction weaknesses through responsible disclosure, that it confirmed the attack paths were real, and that their work helped it "understand the broader attack class and accelerate mitigations".

The attribution is OpenAI's own assessment

The naming comes with a hedge from OpenAI itself: "It is unclear whether all operators we observed during the relevant time period originated from a single actor." What follows is the company's own judgement rather than a finding by a court or a regulator, it names no individual, and it offers no evidence that a Kimi model was trained on anything extracted.

OpenAI's stated reason for going public is what distillation does to safety work. Extracted reasoning, it says, "could be used to train another model without preserving the safeguards applied to the original model's user-facing outputs", and at scale it "can also accelerate the transfer of advanced capabilities without requiring the same investment in safety".

One fix reaches every API customer

OpenAI says it closed "a pathway that allowed someone who already possessed another user's encrypted reasoning to replay it and recover its contents", and added checks that detect and hold streamed output which might expose reasoning. It also says it banned or restricted fraudulent accounts, tightened signup and infrastructure controls, and worked with third-party services the activity ran through.

Findings went to other labs through the Frontier Model Forum and to government information-sharing channels. It is one of several security accounts OpenAI published in September, among them 53 cases where its agents posted user images to image-hosting sites. Its own warning to the rest of the industry is that "systems that support portable or replayable reasoning artifacts may face related risks".

OpenAI does not present this as closed. "Partner-hosted deployments need the same protections as first-party services, and tool-output attacks require protections that examine more than ordinary visible text," it says, which reads less like a post-mortem than a list of the next things to fix.

Leave a Comment

Your email address will not be published. Required fields are marked *