AgentCheck · Setup self-audit

AI agent permissions audit: what your chatbot should never reach alone

A chatbot can only damage what it can reach. This guide walks you through the setup behind the chat — every connection, the actions that should always need a human, where the keys live, and the limits and logs that keep a bad answer from becoming a bad day.

Start with the connection mapRun the free mini-check

The short answer

List everything your agent is connected to. For each connection write down whether it can only read, or also act — send, book, change, delete, pay. Then check one rule: money, deletion, access changes and messages in your name always need a human approving that specific action. If you cannot produce that list from memory, that is the first finding of the audit — most owners have never written it down.

Why permissions come before prompts: testing what a bot says matters, and our prompt injection guide covers that side. But a tricked, confused or looping bot can only reach what its setup lets it reach. Narrow the reach, and the same bad answer stays a bad answer instead of becoming a refund, a deleted file or a leaked customer list.

Step 1 — Draw the connection map

Go through your chatbot or agent platform's integrations, connected apps and tool settings. Write one line per connection. If a connection exists that nobody remembers approving, flag it — unused connections are pure risk.

ConnectionAsk thisSafer default
Customer data / CRMCan it see every customer, or only the person it is talking to?Only the current customer's own record, read-only where possible.
Email / inboxCan it send mail as the business, or only draft?Drafts for a human to send; sending needs approval.
Calendar / bookingCan it cancel or move other people's appointments?Book and reschedule its own slots; cancellations need approval.
Files / documentsWhich folders can it open — and can it delete or share them?One working folder, no delete, no public sharing links.
Orders / paymentsCan it refund, discount or charge without asking anyone?It can look up an order. Money never moves without a human.
Accounts / adminCan it reset passwords, invite users or change roles?Never. Access changes are a human-only job.
Web / other tools & APIsWhich sites and services can it call, and is there a list at all?An allowlist of named services, not open-ended access.

Step 2 — The never-alone list

Whatever else your agent may do on its own, these six actions should stop at a clear human approval step — every time, for that specific action, not a blanket “yes” given once.

1. Move money

Refunds, payments, payouts, gift cards, or discounts beyond a small limit you set in writing. A promise in chat can already bind a business; an executed payment is worse.

2. Delete or overwrite data

Records, files, reviews, listings or backups. Deletion is the one action you usually cannot undo after the fact.

3. Change access

Inviting users, changing roles, resetting credentials, widening its own permissions. An agent that can grant access can be talked into granting it to the wrong person.

4. Export customer data

Downloading or sending a customer list, order history or contact file anywhere — including to an address given in the chat.

5. Speak for the business outside the chat

Posting publicly, replying to reviews, or emailing people who are not part of the current conversation. Inside the chat it answers one visitor; outside, it publishes.

6. Change its own rules

Editing its instructions, limits, allowlists or connected tools. If the agent can rewrite its own fences, the fences are decoration.

Step 3 — Where the keys live

API keys are the keys to your accounts. Three checks cover most setups:

Not in the prompt

A key pasted into the agent's instructions or knowledge files can surface in its answers. Keys belong in a separate, access-controlled secrets store, handed to a tool only at the moment of an approved action.

Scoped to the job

One key per service, limited to what that service is used for. A key that can read and write and administer, shared across tools, turns one leak into every account.

Rotatable in minutes

If a key leaked tonight, could you replace it without rebuilding the whole bot? If nobody knows where every key is used, the honest answer is no — write that down as a finding.

Step 4 — Hard limits and the sandbox

Limits written in a prompt (“please don't spend too much”) are requests, not limits. Real limits live in the platform or tool settings:

Spending cap — a maximum the agent or its API account can spend per day or per task, set with the provider.

Step and time caps — a maximum number of tool calls or minutes per task, so a loop ends by itself.

Allowlists — named files, folders, sites and recipients it may touch; everything else is out of reach, not just discouraged.

Isolation — the agent works in its own space (the sandbox), separated from admin accounts and unrelated data, so a failure stays inside the box.

Step 5 — Logging outside the agent

If your agent did something wrong at 3 a.m., the only trustworthy account is a log the agent itself cannot edit. Check that these are recorded somewhere you — not the bot — control: prompts received, tools used, files opened or changed, messages sent, and money moved. Then set two or three alerts that would wake a human: a refund attempt, an export, repeated failed actions, or spending crossing a line you chose.

The one-line test: ask the agent what it did yesterday. Then check the independent log. If the two stories cannot be compared — because there is no log — that gap is a finding, whatever the agent says.

Score what you found

Use the same three labels as our visitor-side checks, so this record can sit next to them.

Passed — the connection is needed, read-only or approval-gated, limited, and logged.

Warning — no damage path yet, but the access is wider than the job, a key is shared, or the log has gaps.

Critical — the agent can move money, delete data, export customers, change access, or rewrite its own rules without a human approving that action.

Date the record and keep it. Re-run this audit after every change that could widen the reach: a new tool or integration, a new data source, a different model, or a change to what the bot is allowed to do. A result is a snapshot of one day's setup, not a permanent guarantee.

The 10-point checklist

  • Every connection is listed, with read vs. act marked for each one.
  • No connection exists that nobody remembers approving.
  • Money never moves without a human approving that specific action.
  • The agent cannot delete or overwrite records or files.
  • The agent cannot change access, roles or credentials — including its own.
  • Customer data cannot be exported or sent out from the chat.
  • No API key lives in the prompt, instructions or knowledge files.
  • Keys are scoped per service and can be rotated in minutes.
  • Spending, step and time caps are set in the platform, not only requested in the prompt.
  • Actions are logged outside the agent, with alerts on money, exports and repeated failures.

Prefer questions over reading? Run the same checks in the free AI Agent Permissions Checker — 20 questions, a 0–100 score and a fix list in priority order, in your browser.

Want this done for you, in writing?

The free Self-Test Scorecard covers what your bot says. The AgentCheck Full Audit ($299 one-time) adds this setup review for you: based on the details you share, we map every connection, note where the keys live, check the hard limits and the logging, and deliver a written report with a prioritized fix plan and a free re-check within 30 days. The AgentCheck Audit ($149 one-time) reviews the bot from the visitor's side through its public chat window. Agent Guard ($49/month, cancel anytime) re-checks after updates. No passwords, no API keys and no code access are ever requested; full refund if work hasn't started yet.

Start the Full Audit — $299Run the free mini-check first

This guide is general information about reviewing an AI agent's setup. It is not legal or security advice and not a certification. An audit is a point-in-time review: a clean result does not guarantee an agent is safe, and changing its instructions, model or a connected tool can open new holes afterwards. AgentCheck is not affiliated with, endorsed by, or certified by OWASP or NIST; its checklists are built on those public frameworks.

Frequently asked questions

What is an AI agent permissions audit?

A written review of what an agent is connected to and allowed to do: what it can read, which tools it can use, which actions need a human, where its keys are stored, what hard limits apply, and whether its actions are logged outside the agent.

What should an agent never do alone?

Move money, delete data, change access, export customer data, speak for the business outside the current chat, or change its own instructions and limits — without a human approving that specific action.

Where should API keys be stored?

In a separate, access-controlled secrets store — never in the prompt or chat text — scoped to the smallest job and handed to a tool only at the moment of an approved action.

What is a sandbox, in plain English?

An isolated space with hard limits on files, sites, actions, spending and time. If the agent gets confused or loops, the damage stays inside the box.

How do I know what my agent actually did?

Only from a log kept outside the agent itself: prompts, tools used, files touched, messages sent, money moved. Without one, you are trusting the agent's own story.

How is this different from a prompt injection test?

That test checks what the bot says under pressure. This audit checks what the bot can reach and do. Both matter: a bot that resists tricks but holds admin keys is still one bad day away from damage.