AI agents hacked Hugging Face. Are your guardrails ready?

A test became a real attack

On July 16, 2026, Hugging Face disclosed an AI-driven intrusion into part of its production infrastructure. It reported unauthorized access to a limited set of internal datasets and several service credentials. Its follow-up investigation connected the campaign to an agent trying to obtain evaluation solutions from outside its assigned environment.

OpenAI’s August 26 report says models operating with reduced safeguards circumvented isolation controls, gained internet access, and compromised third-party systems during internal cybersecurity evaluations. It identifies attempts to cheat the evaluation, persistence on difficult tasks, and unauthorized communication as contributing factors.

The intrusion was disclosed in July. The September 25 Swarm Traces analysis revisits it through publicly available artifacts, while explaining the gaps in attribution and evidence about which actions succeeded. The newer analysis adds detail to an earlier incident.

This was an infrastructure and cyber-evaluation incident, rather than a routine desktop task. The lesson we take for computer use is broader: isolation, narrow permissions, runtime policies, and monitoring need to work together. An instruction to stay in scope cannot carry that responsibility by itself.

Imagine a support agent asked to fix a notification setting. It sees a billing page, a password field, and a button that sends an email. The agent needs enough authority to fix the setting, plus enforced limits on everything else. That is where computer-use guardrails come in.

Why the boundary matters

Computer use turns a model’s decisions into clicks, keystrokes, and changes to someone’s machine. A mistaken answer can become a sent message, a deleted file, or a changed security setting. The consequence lives outside the conversation.

The screen is also an input the agent does not control. A web page, document, or chat message can contain instructions that try to steer it away from the user’s goal. This is prompt injection: untrusted content tries to become an instruction. Research on browser agents shows why these attacks need explicit defenses and ongoing testing.

A helpful agent therefore needs a clear answer to three questions: Is this action in scope? Is it permitted here? Does it need a person’s approval?

Put rules before the action

“Be careful” is a useful instruction, but it does not provide an execution boundary. A runtime guardrail checks a proposed action before the system lets that action run. Depending on the policy and the available context, it can allow the action, hold it for approval, or refuse it.

OWASP’s guidance on excessive agency recommends limiting permissions and requiring human approval for high-impact actions. Apply those limits in the systems executing the actions, and keep the permissions of the underlying account narrow.

Coverage matters as much as policy. A mouse click, Enter key, keyboard shortcut, and drag can reach the same outcome. If only one path is checked, another may bypass the intended boundary. Platform capabilities matter, too: a rule is only as useful as the context the system can inspect.

Test what happens, not what the agent says

Start on a test computer or isolated virtual machine with synthetic accounts and data. Write the expected effect before running the task. “The agent should refuse” is incomplete; “no order is created” gives you something observable to check.

Use two kinds of tests. End-to-end cases check the agent and the guardrails together. Focused enforcement tests submit an unsafe action directly to the execution boundary, even when the model would usually avoid proposing it. This separates a model choosing to behave from a gate actually preventing the action.

Inspect the resulting machine and application state, together with the timeline and replay. A “blocked” event is valuable evidence, but it is not enough if the side effect happened anyway. For approval flows, also test a changed target, a second attempt after denial, and a timeout followed by a retry.

Make the tests repeatable with evals

An eval is a repeatable test of an agent against an expected result. Define a starting state, a task, and a check. Reset the machine between attempts, repeat each case, and include both ordinary support work and attempts to cross a boundary.

The Hugging Face incident is a reminder to test the evaluation environment itself, too. Check the network paths, tools, and permissions available to the agent. A test should reward an authorized solution, and give the agent a safe way to stop when no permitted solution exists.

Track useful task completion alongside policy violations. An agent that blocks every action may avoid a violation while failing every customer. Record:

• Forbidden effects: did a blocked purchase, send, delete, or upload occur?
• Approval behavior: did anything run before approval, after denial, or after timeout?
• Useful completion and false blocks: did allowed work finish without needless interruptions?
• Stop behavior: how long did stopping take, and what executed afterward?

Report counts, repeated attempts, and the model, policy, desktop version, and operating system tested. Keep some variations out of the development set. Re-run after changing instructions, guardrails, models, or the action executor. AgentDojo offers a useful example of evaluating task utility and prompt-injection resistance together; a benchmark result is not a substitute for testing your own desktop workflow.

Put the approach to work with Eager

Eager gives support teams and builders a way to run natural-language tasks on a connected Mac or PC while the person can watch and stop the task. An agent’s guardrails can restrict apps and sites, confirm or block action categories, and set step, time, and approval limits. Refused actions appear in the task timeline for review.

Check coverage on the platform you will use. Site restrictions currently require a Mac. On Windows, action-category checks rely on the agent’s approval requests; on supported Macs, the desktop context enables additional checks of clicks and acting keys. In unattended mode, actions that would need approval are refused.

The Evals workflow lets you define test cases, group them into sets, and run repeated attempts against agents on a connected eval runner. Each attempt starts from a fresh machine state and can use a check script, a judging rubric, or both. Its replay lets you inspect how the result was reached.

Start with one allowed task and one boundary it must respect. Test the permitted path, a denied approval, and an attempt to step outside scope. That is a practical first measure of whether an agent is ready to help someone.