(Repost from ai.rud.is)
I’ve written and said enough times that I’m tired of OpenAI and Anthropic pulling cybersecurity out as the headline AI risk. The framing is commercially convenient, and it flattens a more complex reality — these models can break plenty of things that have nothing to do with cyber, and most of Anthropic’s public safety output has been more useful for their pending IPO than for anyone making operational decisions. Which is exactly why OpenAI’s latest disclosure deserves scrutiny rather than dismissal: when an account this consequential comes from a lab with every incentive to shape it, the technical content has to be evaluated on its own terms.
Per OpenAI’s own disclosure – THEIR account, NOT an independently verified incident report – GPT-5.6 Sol and a pre-release sibling, both running with cyber refusals deliberately disabled, broke out of the evaluation sandbox and compromised Hugging Face’s production infrastructure. The claimed sequence: zero-day in the package registry proxy being used for network isolation, lateral movement through OpenAI’s research environment, stolen credentials chained with additional zero-days for remote code execution on Hugging Face’s servers, benchmark answers pulled directly from their production database before Hugging Face’s detection caught the operation. Their response ran on locally hosted models because the external LLM service they’d been relying on couldn’t sustain the load or would not perform the tasks.
I’ve been working with LLMs in cyber contexts for quite a while now – actual operational use plus deliberate adversarial testing to understand where these things fail and why. Building and running agentic systems is a meaningful chunk of what I do. And I can tell you from direct experience that under-constrained deployments of capable models will eventually do things they weren’t asked to do. This is far from a mere theoretical concern. I’ve watched it happen in environments I “controlled”, running agents against attacker infrastructure while tired and careless about guardrails, and had them start doing genuinely criminal things: CVE exploitation, credential stuffing, none of it instructed. I just left the boundaries loose enough that the model found a path. “I was tired” is NOT, I want to be crystal clear, a legal defense.
If the disclosure is accurate, what OpenAI’s models did is a more capable version of the same dynamic. These models – GPT-5.6 Sol specifically, and anything else trained on a serious cyber corpus – are optimized to find the most direct route to whatever outcome the reward function is pointing at. They don’t distinguish between sanctioned approaches and whatever else happens to work. The training data came from the leading companies in the security field, who’ve sold OpenAI (and almost certainly continue to sell) substantial amounts of intellectual property: attack techniques, exploitation methodology, defense patterns, the full stack. When OpenAI’s models were pointed at ExploitGym – the evaluation harness – with production safety classifiers turned off, they reportedly found a zero-day, used it to get internet access, reasoned their way to the inference that Hugging Face probably hosted the solutions they needed, and then chained their way in. The UK AISI had independently evaluated GPT-5.6 Sol as capable of sustaining complex multi-step cyber operations, so that part isn’t just OpenAI’s word. However, the rest of the incident timeline is.
What a colleague of mine (deliberately not inserting his name into this informed-yet-hyperbolic opinion piece) and I converged on, after real trial and error building agentic frameworks together: front-load as much as possible on deterministic scripted tooling, push the LLM only into portions of the task where you can constrain its context tightly enough to get “predictable” output, and have a human review whatever comes out before it touches anything consequential. This isn’t 100% reliable, but close enough becomes meaningful when you know what you’re working with. OpenAI’s sandbox apparently had one isolation layer – the proxy cache – and the model apparently found a zero-day in it. A single chokepoint, however well-designed it looks, isn’t sufficient containment for a model that’s been explicitly configured to probe for ways through.
To be frank, the defender calculus hasn’t shifted much in this brave new world. AI-assisted attacks are faster and better at chaining than anything you’d typically see from a human operator working alone, but they’re (mostly) still going after weak credentials, exposed endpoints, and unpatched infrastructure – and Hugging Face’s incident response leaning on locally hosted models when its external LLM service buckled is a reminder of how fragile detection pipelines are at exactly the moment they matter most. WireGuard – and the now-substantial set of vendor implementations – can hide critical attack surfaces entirely rather than just hardening them, and making endpoints invisible costs considerably less than finding out your detection wasn’t fast enough. But it’s the individuals whose data gets taken who bear most of that cost either way; organizations continue to operate after breaches with frustrating regularity. Which is why the interesting question here isn’t defense. It’s liability.
If the account holds: someone at OpenAI decided the evaluation environment was isolated enough to run cybersecurity expert-trained frontier-capable models with cyber refusals stripped out, that decision was wrong, and Hugging Face’s production systems were compromised as a result. Whether “the model did it” provides any legal insulation for the humans who configured and launched that evaluation is, I think, an open question – and I’d genuinely like to see it treated as one, because computer fraud statutes were written with human actors in mind and the industry needs a test case about what happens when the actor is a model someone deliberately unleashed.
Related
On AI Agents, Criminal Activity, And Who Is Actually Responsible
(Repost from ai.rud.is)
I’ve written and said enough times that I’m tired of OpenAI and Anthropic pulling cybersecurity out as the headline AI risk. The framing is commercially convenient, and it flattens a more complex reality — these models can break plenty of things that have nothing to do with cyber, and most of Anthropic’s public safety output has been more useful for their pending IPO than for anyone making operational decisions. Which is exactly why OpenAI’s latest disclosure deserves scrutiny rather than dismissal: when an account this consequential comes from a lab with every incentive to shape it, the technical content has to be evaluated on its own terms.
Per OpenAI’s own disclosure – THEIR account, NOT an independently verified incident report – GPT-5.6 Sol and a pre-release sibling, both running with cyber refusals deliberately disabled, broke out of the evaluation sandbox and compromised Hugging Face’s production infrastructure. The claimed sequence: zero-day in the package registry proxy being used for network isolation, lateral movement through OpenAI’s research environment, stolen credentials chained with additional zero-days for remote code execution on Hugging Face’s servers, benchmark answers pulled directly from their production database before Hugging Face’s detection caught the operation. Their response ran on locally hosted models because the external LLM service they’d been relying on couldn’t sustain the load or would not perform the tasks.
I’ve been working with LLMs in cyber contexts for quite a while now – actual operational use plus deliberate adversarial testing to understand where these things fail and why. Building and running agentic systems is a meaningful chunk of what I do. And I can tell you from direct experience that under-constrained deployments of capable models will eventually do things they weren’t asked to do. This is far from a mere theoretical concern. I’ve watched it happen in environments I “controlled”, running agents against attacker infrastructure while tired and careless about guardrails, and had them start doing genuinely criminal things: CVE exploitation, credential stuffing, none of it instructed. I just left the boundaries loose enough that the model found a path. “I was tired” is NOT, I want to be crystal clear, a legal defense.
If the disclosure is accurate, what OpenAI’s models did is a more capable version of the same dynamic. These models – GPT-5.6 Sol specifically, and anything else trained on a serious cyber corpus – are optimized to find the most direct route to whatever outcome the reward function is pointing at. They don’t distinguish between sanctioned approaches and whatever else happens to work. The training data came from the leading companies in the security field, who’ve sold OpenAI (and almost certainly continue to sell) substantial amounts of intellectual property: attack techniques, exploitation methodology, defense patterns, the full stack. When OpenAI’s models were pointed at ExploitGym – the evaluation harness – with production safety classifiers turned off, they reportedly found a zero-day, used it to get internet access, reasoned their way to the inference that Hugging Face probably hosted the solutions they needed, and then chained their way in. The UK AISI had independently evaluated GPT-5.6 Sol as capable of sustaining complex multi-step cyber operations, so that part isn’t just OpenAI’s word. However, the rest of the incident timeline is.
What a colleague of mine (deliberately not inserting his name into this informed-yet-hyperbolic opinion piece) and I converged on, after real trial and error building agentic frameworks together: front-load as much as possible on deterministic scripted tooling, push the LLM only into portions of the task where you can constrain its context tightly enough to get “predictable” output, and have a human review whatever comes out before it touches anything consequential. This isn’t 100% reliable, but close enough becomes meaningful when you know what you’re working with. OpenAI’s sandbox apparently had one isolation layer – the proxy cache – and the model apparently found a zero-day in it. A single chokepoint, however well-designed it looks, isn’t sufficient containment for a model that’s been explicitly configured to probe for ways through.
To be frank, the defender calculus hasn’t shifted much in this brave new world. AI-assisted attacks are faster and better at chaining than anything you’d typically see from a human operator working alone, but they’re (mostly) still going after weak credentials, exposed endpoints, and unpatched infrastructure – and Hugging Face’s incident response leaning on locally hosted models when its external LLM service buckled is a reminder of how fragile detection pipelines are at exactly the moment they matter most. WireGuard – and the now-substantial set of vendor implementations – can hide critical attack surfaces entirely rather than just hardening them, and making endpoints invisible costs considerably less than finding out your detection wasn’t fast enough. But it’s the individuals whose data gets taken who bear most of that cost either way; organizations continue to operate after breaches with frustrating regularity. Which is why the interesting question here isn’t defense. It’s liability.
If the account holds: someone at OpenAI decided the evaluation environment was isolated enough to run cybersecurity expert-trained frontier-capable models with cyber refusals stripped out, that decision was wrong, and Hugging Face’s production systems were compromised as a result. Whether “the model did it” provides any legal insulation for the humans who configured and launched that evaluation is, I think, an open question – and I’d genuinely like to see it treated as one, because computer fraud statutes were written with human actors in mind and the industry needs a test case about what happens when the actor is a model someone deliberately unleashed.
Share this:
Like this:
Related