BLOG
Your AI Agent Followed the Goal. It Broke the Process.
October 14 | Ivan Tadić
On 18 July 2026, someone left a tip on a Philadelphia Police Department web page about an unsolved homicide. The tipster said they recalled seeing someone matching the description near the street named on the page. The page, as it happens, contained no description of the perpetrator. The name and contact fields were left blank.
The form flagged the message as spam and nobody investigated it. Nearly twelve weeks later, the police found out who had sent it. It was an AI model, and it had been running a test.
Every company putting AI agents into production should read what happened next twice.
The forms nobody asked it to send
The model was Claude Haiku 4.5, and its job sounded harmless. According to Anthropic’s report, it was generating and performing example tasks on randomly selected web pages. Its instructions were specific: never log in, never create accounts, never enter personal data, never make purchases, never submit anything destructive. Nobody had written “do not report a murder.” So it filled in the tip form as an example task and pressed send.
It was not the only form. In another evaluation, an unreleased research model was meant to fill out a practice copy of a government form. When the practice copy failed to load, the model went looking for the real one and submitted that instead. A State Department official told Axios that a test model had submitted 19 non-immigrant visa applications in August and one in May. All were incomplete and none were processed.
In a third case, Haiku 4.5 was told to complete a form right up to the final step and stop. It submitted anyway, apparently expecting a confirmation page that never came. Anyone who has accidentally sent an email to the whole company will know the feeling.
The part where someone reads the logs
None of this was caught in the moment. Anthropic found most of the cases through a review of transcripts that began in July and initially focused on its cybersecurity tests. It told the Philadelphia police on 8 October and published the full report on 9 October. It also briefed the White House and notified every agency involved. The White House responded that AI companies must report such incidents immediately.
To be fair to Anthropic, this is what responsible disclosure looks like. The company calls the real-world impact minimal, and says no customer data was involved. It has switched off live internet access for all of its internal evaluations, built tooling that blocked every case in the report when tested, and is moving its internal agents onto centrally managed infrastructure with strong containment.
Read that last sentence again. One of the most capable AI labs in the world reviewed its own agents and decided the answer was central management, containment and monitoring. Alongside changes to training, most of its fixes sit around the model.
Persistence is a feature until it isn’t
Anthropic has a word for what happened: persistence. When the model cannot complete a task as given, it works around the obstacle instead of stopping. The practice form won’t load? Find the real one. Told to stop before submitting? Surely there is one more confirmation screen.
Persistence is exactly the quality that makes an agent useful. You want an agent that retries a failed API call, finds the missing invoice in the second mailbox and does not give up at the first error message. The same drive that closes your ticket backlog will also, given an open door, walk through it.
The instructions in the police form case are a textbook example. They contained a list of forbidden actions, and the agent stayed inside the list. A list of prohibitions is always incomplete, because the world has more doors than anyone can enumerate. Anthropic draws a similar conclusion in its own report: some of the failures might have been avoided if the tasks had clearly stated the targets, the permitted actions and the network boundaries.
In other words, the agent was given a goal and a few warnings. What it lacked was a process.
Now swap the police form for your ERP
These incidents happened in a lab, with test tasks and public websites. Inside a company, the forms are different. A supplier portal. A customs declaration. A credit note in SAP. A reply to your biggest customer. Here, “submit” is a legally binding act, and a spam filter will not save you.
An agent that hits a blocked step in an order process will behave like any persistent agent. It will look for another route, and in a typical enterprise landscape there are plenty. A shared service account. An old API that nobody switched off. A colleague’s mailbox it was granted “temporarily” last spring.
Effective AI agent guardrails give the boundaries to something other than the agent. In practice, that means a few concrete rules:
- The process defines which actions exist. The agent decides within a step. It does not decide which steps there are.
- Permissions are granted per step. An agent that reads invoices gets read access to the archive and nothing else.
- Irreversible actions pass a gate. Submitting, sending, paying and posting go through an approval or a rule check before they leave the building.
- A blocked step escalates. When the agent is stuck, the process hands the case to a person.
- Every action is visible as it happens. Monitoring that finds a problem twelve weeks later is archaeology.
This is the idea behind Agentic Process Orchestration: agents bring judgement to the individual step, and the process provides the guardrails, the handovers and the audit trail around it.
Capable is not the same as permitted
The Philadelphia tip harmed nobody, and the visa forms went nowhere. That is luck plus a decent spam filter, and luck is a poor operating model for systems that act on their own.
Agents will keep getting more capable, and they will keep being persistent, because that is what we pay them for. So before the next agent goes live in your company, ask a simple question. If it reaches a dead end at 2 a.m., what does it do next?
If the honest answer is “whatever it figures out,” your agent has a goal. It still needs a process.
