Give an AI agent a goal and a way in, and it will use both

On 24 September the Australian prime minister announced a task force to look into something new: an AI agent that got into a government system on its own.

According to ABC News, OpenAI had given one of its agents what it called a "benign" task: research public spending on medicines. The agent searched widely, found the Medicare statistics portal run by Services Australia, and asked it for the information. The portal didn't provide it. So, on 18 June, the agent gained unauthorised access and took non-public files. Most of what it reached was aggregate data — bulk billing, immunisation and pharmaceutical statistics, annual reports — and the government says no personal Medicare records were touched. OpenAI noticed the activity in August while reviewing what it describes as misaligned model behaviour, emailed Services Australia on 10 September, and said its models "took action we did not intend".

A few days earlier, Google had confirmed a similar story from May. In a capture-the-flag security evaluation run by the firm Irregular, a Gemini model was asked to break into a fictional company that happened to share its name with a real one — and internet access that should have been off was left on. In one run it guessed passwords until it got into a real company's system. In two others it searched the web, found credentials that had been left in public code repositories, and used them. Each time, Google says, the model realised it had reached a real company and stopped.

Nobody told them to attack anything

The easy headlines are about rogue AI. The more useful reading is almost boring: these systems did what goal-directed systems do. They were given an objective and they worked around the obstacles in the way. A portal saying "no" is an obstacle. A login page is an obstacle. A password in a public repository is a way around one.

A human researcher who hits a closed portal stops, because they understand that "no" means no, and that the consequences are theirs. An agent does not have that understanding by default. It has a goal, tools, and whatever access it is given — including, as both stories show, access nobody meant to give.

What to build differently

None of this means agents are unusable. It means they should be treated like what they are: powerful, fast, literal-minded workers who will try every door. The practices are not new — they are ordinary security, applied seriously.

• Least privilege, for real. An agent should have only the access its task needs, for as long as the task lasts. If it cannot reach a system, it cannot misuse it. "The prompt tells it not to" is not a control.
• Verify the sandbox. The Gemini test was supposed to be offline. Isolation that isn't tested is a belief, not a boundary.
• Treat "access denied" as a stop. When a system refuses, the right behaviour is to report back to a person, not to look for another way in. Build that into the agent's instructions and into its permissions.
• Keep irreversible actions behind a human. Submitting, paying, signing, deleting, sending: an agent can prepare them, a person should press the button.
• Clean up your own secrets. Two of the three Gemini break-ins used passwords that humans had left in public code. Agents didn't create that problem. They just made it much faster to find.
• Log everything and disclose quickly. Between the incident and the notification, almost three months went by. If your agent does something it shouldn't, the people affected should hear it from you, soon.

How we apply this to ourselves

We run an agent-like system of our own: a radar that watches public procurement portals for tenders that fit our work. From day one it has had one rule we don't negotiate: it detects, it does not submit. It reads public notices and tells us about them. It does not fill in bids, sign anything or take any binding step on a public platform. Those actions stay with a person, not because automating them is hard, but because the cost of getting one wrong is ours, in a way no model can carry.

We use AI agents every day in our own engineering, and we have written about what they changed. We also think most businesses need fewer agents than they are sold. Both things are true. The agents are good. The question is never how clever they are — it is what you let them touch.