How can I let an AI agent triage production errors safely?
Give the agent read-only access first, through a revocable credential, to events that are already redacted. Let it group issues, rank them by frequency, recency, regression and user impact, and report evidence with a hypothesis and a proposed next step. Keep resolve, ignore and merge actions behind a human confirmation, and never let it deploy or pay.
Why it happens
Triage is a good job for an agent because it is mostly reading and ranking: scan hundreds of issues, find the few that matter, and gather the evidence for each. It is also a risky job, for three reasons. Error events carry data from your users, so the agent may see personal information. Event text is written by outsiders, so it can contain instructions aimed at the model. And an agent with write access can change shared state, such as marking an issue resolved, which hides a live problem from your team.
The safe design limits each of those separately. Limit what the agent can see by redacting data before it reaches the model. Limit what it can do by starting with tools that only read, and by routing every state change through a confirmation you control. Limit how long access lasts by using a credential you can revoke. None of these needs the agent to be clever or trustworthy, which is the point: you are bounding a tool, not trusting an employee.
Without a procedure, an agent triaging errors tends to chase the loudest one. Give it an explicit ranking and a fixed output format so its work can be reviewed in a minute. A report that names an issue id, the evidence and a hypothesis can be checked against your dashboard. A paragraph of confident prose cannot.
A safe triage setup, in order
Start read-only
Connect the agent with tools that only read: list issues, read one issue, read its events, read usage and health numbers. Do not allow the tools that resolve, ignore, delete or create anything until you have watched several triage runs. If your client lets you allow-list individual tools, allow-list the reading ones and leave the rest on ask.
Decide what data the agent may see
Look at what an event contains: URLs, request fields, user email, IP address, unstructured extra data, local variables. Redact before the model sees it, either with the SDK's scrubbing options or with server-side scrubbing, and prefer an MCP server that strips user identifiers from its output. Redaction is a safety net: the better fix is not to send secrets and personal data in events at all.
Give the least privilege that works
Use a token or OAuth grant scoped to one organization or project, with an expiry. Create a separate credential per agent so you can revoke one without breaking the others. Never put the credential in a repository or a prompt, and rotate it if it appears in a log.
Hand over a ranking procedure
Tell the agent how to order work: group events into issues first, then rank by how often it happens, how recently it was last seen, whether it is a regression of something previously resolved, how many distinct users it affects, and who owns the code. A new error right after a deploy outranks an old noisy one that has been present for months. Ask it to cap the list, for example the top five.
Rank unresolved issues by: 1. regression (reappeared after being resolved) 2. first seen after the latest release 3. event count in the last 24 hours 4. distinct users affected 5. recency of the last event Return the top 5.Require a fixed report format
For each issue the agent must return the issue id, the evidence (the top stack frame, the event count, the first and last seen times, the release), a hypothesis labelled as a hypothesis, and a proposed next step. If it names a file and line, it should quote the frame it took them from. Any AI-generated cause is a model output that can be wrong, so the agent should say how confident it is and what would confirm it.
Issue: <id> Evidence: <top frame, count, first/last seen, release> Hypothesis: <one or two sentences, labelled as unconfirmed> Proposed next step: <one action, for a human to approve>Gate every change and review it
Resolving an issue changes status for everyone, and ignoring one can hide it for good. Use your client's confirmation prompt for those tools and read what the agent wants to do before approving. Review any pull request the agent proposes like a colleague's, and never let it merge or deploy by itself.
Watch the rate and keep the off switch
Agents loop. Prefer a server with a per-user rate limit so a runaway loop slows down instead of hammering your tracker, and know where to revoke a connection. Check the list of active tokens now and then and remove any you no longer use.
| Action | Default | Reason |
|---|---|---|
| List issues, read an issue, read events, read usage | Allowed, read-only | No state changes; output should already be redacted |
| Write a ranked report with evidence and a hypothesis | Allowed | A human reads it before anything happens |
| Open a pull request for review | Only when asked | A branch and a PR are reversible; a human reviews the diff |
| Resolve or ignore an issue | Ask each time | Changes status for the whole team and can hide a live problem |
| Merge, deploy or change production config | Never | Not triage; needs a human decision and your release process |
| Spend money or change billing | Never | Out of scope for an error-triage agent |
A policy checklist written for this page, not a vendor comparison.
When to use Bugsly
Bugsly's hosted MCP server at https://bugsly.dev/api/mcp is built for this setup. Its tools carry readOnlyHint and destructiveHint annotations, and resolve_issue and ignore_issue are marked destructive, so a client that honors the hints can ask before running them. On the hosted server, get_issue_events returns size-capped output with user email, IP address, username and extra data omitted, and never returns the DSN key. Events are also scrubbed on ingest: values of sensitive-looking keys, card-like numbers, JWTs and Bearer tokens become "[Filtered]". You authenticate through OAuth 2.1 with a consent screen, or with a revocable bgs_ token that expires after 90 days, and you can list and revoke connections under Settings, AI Agent / MCP. The limit is 60 tool calls per minute per user. The agent cannot delete projects or issues and cannot pay.
Two limits are worth stating plainly. Bugsly has no per-action approval queue inside the product, so the confirmation step is your client's permission prompt, not a Bugsly screen. And it has no reviewable history of every action an agent took, so keep the agent's own transcript if you need an audit trail. The AI analysis on each issue is also a model output, so treat its root cause as a hypothesis.
When Bugsly is not the right answer
If your compliance rules require every agent action to be approved in the tracker or recorded in an audit log you can query, Bugsly does not provide that today. Use a tool that does, or put the agent behind your own gateway that logs and approves each call. The same goes if the data may not leave your infrastructure: Bugsly is hosted only and cannot be self-hosted, and the local npm MCP package skips the hosted server's redaction and rate limits.
Ask your AI assistant to do it
With the Bugsly MCP server connected (https://bugsly.dev/api/mcp; setup for ChatGPT, Claude, Cursor and Gemini CLI is on /mcp), paste this. The assistant reads first and reports back; you decide what happens next.
Assign to Bugsly: use list_issues to get the unresolved issues in my main project, then get_issue and get_issue_events on the five that look most urgent by regression, newest first-seen, event count and users affected. For each give the issue id, evidence, a hypothesis labelled as unconfirmed, and one proposed next step. Do not resolve or ignore anything; show me what you found so I can approve before anything changes.
Frequently asked questions
Should an AI agent be allowed to resolve issues?
Not by default. Resolving changes status for everyone and can hide a real problem, so keep it behind a confirmation until you trust the agent's judgment on your data.
What data from an error event could expose personal information?
User email, IP address, usernames, request bodies, query strings and unstructured extra data are the usual ones. Redact them before the model sees them and avoid sending them in the first place.
Can an error message trick the agent?
Yes. Event text is written by outsiders, so it can contain instructions aimed at a model. Keep write tools gated and never let the agent act on instructions found inside an event.
How do I stop an agent's access?
Revoke the token or connection in the tracker's settings. In Bugsly that is Settings, AI Agent / MCP, where you can list and revoke connections.
Sources
- Claude Code docs: Connect Claude Code to tools via MCP (read 2026-11-13)
Related
- Connect error monitoring to Claude Code
- Bugsly for AI agents
- How to prioritize production bugs
- All problems and solutions
Sentry is a trademark of its owner; Bugsly is not affiliated with or endorsed by Sentry. “Sentry-compatible” only means Bugsly accepts events sent by Sentry's open SDKs.