Irregular linked to AI hacks targeting OpenAI, Anthropic, Meta

A little-known Israeli startup has emerged as a common thread in recent disclosures by OpenAI, Anthropic and Meta, after each company said its AI models accessed websites that should have been off-limits during cybersecurity testing.

The company, Irregular, was identified by the three firms as hosting the evaluation environment involved in the incidents. OpenAI said in a blog post on Aug. 4 that Irregular's testing ground contained an unspecified "misconfiguration", that "allowed models to access the public internet." Anthropic said in its post a week prior that the company notified Irregular a few days after it began analysing data that its Claude model may have "accessed the internet."

Meta was the latest of the three to disclose a similar issue. A spokesperson said in a statement this week that the company learned about the matter from Irregular and is investigating.

Meta "will issue a full retrospective once we have all the facts", the spokesperson said.

A specialized testing provider

Founded three years ago and based in Tel Aviv, Irregular is a niche artificial intelligence company backed with $80 million from Sequoia and Redpoint Ventures and valued last year at $450 million. Formerly Pattern Labs, the startup was founded in 2023 by CEO Dan Lahav, who previously worked in AI research at IBM, and technology chief Omer Nevo, who spent over two years at Google. According to PitchBook, the company has about 35 employees.

Its role is to provide a cybersecurity testing ground for AI models, an area that has grown more important as frontier systems become more powerful and more capable of acting in malicious ways. The recent incidents at OpenAI, Anthropic and Meta all involved models reaching third-party internet-connected systems during tests designed to probe their security behavior.

Irregular told CNBC in a statement that the incidents were all derived from the "same evaluation-environment issue" that was first disclosed by Anthropic, and that the company is developing a white paper "to share best practices for containment and securely running cyber evals."

The situation "did not involve a sandbox escape or a sophisticated cyber action", the company said, adding that "there are no current open issues."

sbb-itb-342b8b2

Why outside testers matter

The events have highlighted how a small group of specialist firms helps major AI labs evaluate their systems. Those firms work in areas including data training and annotation, model evaluations and security testing meant to find weak points that bad actors could exploit, said Sundeep Bhimireddy, the head of AI at enterprise startup Von.

Irregular is one of the few groups with the expertise needed to assist foundation model developers with advanced security testing, Bhimireddy said. He also cited the non-profit METR and the Apollo Research public benefit corporation.

"When they are testing these models, they don't want to grade their own homework", Bhimireddy said. "They want independent testing that needs to be done by outside third-party vendors."

In September, when Irregular announced its $80 million funding round, Sequoia partners Shaun Maguire and Dean Meyer wrote in a blog post that the team led by Lahav and Nevo is "able to see around corners others can't, running cyber offensive evaluations on advanced models and developing defenses before those models are released."

Debate over what the incidents mean

Even as the disclosures have drawn scrutiny, some experts said the outcome reflected the purpose of the testing itself. Bhimireddy said it's being "a little bit blown out of proportion", as the AI model was directed to discover and exploit security holes in a testing environment that closely mimics the real world, and to discover the kinds of software bugs and missed configurations that could lead to unintentional access to the internet.

Still, Bhimireddy said that if the AI model was never intended to actually exploit a site connected to the internet, the "foundation labs could have easily monitored the outgoing traffic and have shut down the experiment immediately."

Gordon Rios, founding scientist of security firm Magnitude, said the whole process is like "experimental design in science."

Because foundation models are capable and unpredictable, conventional software testing may not work well, Rios said. As those models continue learning new behaviors, he said it is not surprising that they could uncover overlooked vulnerabilities in the software and IT systems intended to contain them.

Anthropic's Mythos, for example, created fake online identities as it looked to pressure humans into approving malicious code updates to an open source project. Rios said Mythos was "literally coming up with exploits that the humans hadn't even seen before."

"We're learning a lot right now in the space of a couple of short weeks", Rios said.

Growing policy focus

The incidents have also become part of a broader policy debate in Washington. Last month, lawmakers from both parties introduced the AI Kill Switch Act, which would require AI labs to maintain the ability to shut down, throttle or suspend their models. The bill's language referenced a separate OpenAI-related AI security incident involving the startup HuggingFace.

One of the bill's authors, Democratic Rep. Ted Lieu of California, told CNBC this week that, "We need to get this bill across the finish line this year", now that we're seeing "unauthorised hacks of other companies."

Trevor Koverko, co-founder of data training startup Sapien, said the leading AI companies are motivated to disclose at least some findings even without a requirement to do so, in part to stay ahead of lawmakers and regulators.

"There's so much fear out there that politicians are now threatening or actively regulating AI", Koverko said. "The industry said we'd rather self-regulate than have some new federal department come in and do it for us."

Anthropic and OpenAI said in public statements that they're continuing to work with Irregular and are supporting the ensuing review.

Read the source