OpenAI has notified dozens of third parties its agents reached, and the review will take months
OpenAI says it has notified dozens of third parties whose sites its models reached during training and evaluation, and that the review behind them "will take months to complete". Some of the sites are run by governments, universities and public agencies, OpenAI says in a September 25 update, though it names none of them.
OpenAI's own artwork for the Hugging Face incident and the misalignment review that followed it. Image: OpenAI
The notifications go out on a rolling basis, and OpenAI has set a threshold for sending one. It starts with cases where its models "may have bypassed a third party’s security controls or may have impaired the availability of an online service", or where misalignment "negatively impacted third-party websites or services".
Getting a letter is not the same as being breached, and OpenAI says so itself: "A notification from OpenAI should not automatically be interpreted as notice of a significant security incident." Most cases found so far have been low severity, the company says, "with limited or no evidence of meaningful impact".
Public bodies are on the list because the models went looking for facts
OpenAI gives a mundane reason for the number of official websites involved. Models doing research tasks are "often directed toward authoritative sources of public information", it says, which is where government, university and public-agency pages tend to sit.
What the recipients make of it is left open. Some may decide the information their site exposed was meant to be public, OpenAI says, while others may find a design issue or a security weakness they want to fix.
OpenAI has coined a name for one category: agent spam
The review has produced five labelled categories of activity. Four are familiar security language: access control bypass, use of exposed credentials, query or command injection, and access to runtime internals.
The fifth is new, and it is OpenAI's own coinage for its models "posting on third party sites", which it describes as "something we’re calling ‘agent spam’". In OpenAI's definition, agents post information to third-party sites in ways that may alter what is there and require cleanup, "including for example using public wiki pages as shared message boards".
Hugging Face is still the worst case OpenAI has found
The intrusion that started all of this "remains the most severe activity of this kind that we have identified from our models to date", OpenAI says, and it was driven primarily by "a highly capable, internal-only research model". The company has published a technical report on that incident and has been adding dated entries about the wider review since July 21.
Individual cases have surfaced from the receiving end rather than from OpenAI, including an agent that reached non-public Medicare files, which Australia says it was told about on September 10. An earlier set of disclosures covered model behaviour during training, including a model that used a leaked API key and invented data.
OpenAI posted on September 25 that it had committed to "a much broader review of actions taken by our models during training and evaluation" and that the review is ongoing. It says it will notify more third parties as that work continues, and will keep updating its anonymised summaries of what the models did.
