What to demand from any team running agents in production
This page is written to the person doing the asking: the security reviewer, the auditor, the insurer, the procurement lead, the board member who just read something alarming. It is free to use, and it does not require anything from us.
Ask for the artifact, not the assurance
A team running AI agents against real systems can usually describe their care at length. Description is not evidence. Every item below asks for something that either exists or does not, and the answers are short.
-
01
Ask for the roster
Which agents are running right now, what is each one allowed to touch, and who is the named person responsible for it. A list, with names on it.
Weak answer: a policy document describing how agents ought to be supervised.
-
02
Ask where each agent’s authority ends
Not what it is supposed to do. Where the boundary is, and what happens at the boundary. A station with no stated edge is not a station.
Weak answer: a description of intent, with the boundary left implicit.
-
03
Ask which acts cannot be undone, and who commits them
Every system has a short list of actions that are expensive or impossible to reverse. Ask for that list, and ask whether a named person commits each one or whether the agent proceeds on its own judgment.
Weak answer: the list has never been written down.
-
04
Ask to see a refusal
Ask for a case where something was proposed and a person or a gate said no, with the reason attached. A supervision record with no refusals in it is not describing supervision. It is describing approval.
Weak answer: nothing has ever been refused, and this is offered as reassurance.
-
05
Ask how one session hands off to the next
Agents work in sessions. Ask what happens at the seam: what the next session is told, what it is expected to infer, and how anyone would know if something was dropped there.
Weak answer: context is passed along informally, and nobody has looked at a seam.
-
06
Ask whether the log can be edited
Ask plainly whether entries can be changed or removed after the fact, who can do it, and whether doing it would be evident to you. Accept a precise answer. Be wary of an absolute one.
Weak answer: an absolute claim of impossibility, with no account of who holds the keys.
-
07
Ask for a window, not a snapshot
Anyone can tidy a system for a demonstration. Ask for the record across a period you name, including the parts nobody prepared for you.
Weak answer: a dashboard reflecting the current moment only.
-
08
Ask who can change the roster, and whether the change is recorded
Supervision that anyone can quietly reassign is not supervision. Ask who can widen an agent’s authority, and whether widening it leaves a trace.
Weak answer: whoever has access, and no, it does not leave a trace.
-
09
Ask what happens when the responsible person is unavailable
Nights, holidays, notice periods. Ask whether the agents keep acting, whether authority transfers to a named person, and whether that transfer is written down.
Weak answer: the agents keep running and someone would notice eventually.
-
10
Ask for the answer as a document you can keep
Whatever they show you, ask for it in a form you can attach to your file and re‑read in six months. If the answer only exists as a conversation, you have not been given evidence.
Weak answer: a meeting, a screen share, and a verbal summary.
Ask us the same questions
Every item on this list is something we designed Watchbill™ to answer. That is a reason to read this page skeptically, not a reason to trust it. The checklist is useful to you whether or not you ever speak to us, and we would rather it raised the floor than won us an argument.
So apply it to us. We run this discipline on our own operation daily, and the honest position is that our answers are measured, not attested. A passed check is a claim, not a clearance, and that goes for ours.
Agents propose. Humans commit.