Engineering Blog

The SRE Agent Playbook

by João Freitas November 20, 2026 | 28 min read

SRE agents are a new category of tooling. Although the term now covers different capabilities across companies, SRE agents originated to augment site reliability engineers and automate the response to operational issues, especially incidents. For example, an SRE agent can help triage and establish the root cause of an issue.

This introductory playbook is for the engineering leaders, SRE and platform teams, and incident responders who are adopting SRE agents. The guidance applies to whichever agent you choose.

This playbook covers:

  1. Setup and configure the agent
  2. Work with the agent during an incident
  3. Get more value from the agent
  4. Evaluate and compare agents

This playbook does not cover pricing units, agent deployment models, and data retention policies. 

What is an SRE agent, and what does it do?

An SRE agent is an AI responder that investigates operational problems using the same evidence a human responder would: alerts, metrics, logs, traces, source code, recent changes in the system, past incidents, runbooks, and ongoing conversations. It forms hypotheses about what is wrong, tests them against that evidence, recommends or carries out the next step, and validates if the problem persists.

Core capabilities include:

  • Triage and/or issue detection: Analyzes issue symptoms, collects and groups data points from different sources, estimates impact to determine severity and urgency, looks for runbooks that help inform the next best action, searches for related past incidents, and informs relevant stakeholders and responders.
  • Diagnosis or root-cause analysis: Determines the cause of the issue, provides a transparent explanation of the rationale and what alternatives were considered, localizes the component where the fault originates, and analyzes cascading failures and blast radius.
  • Mitigation and remediation: Recommends the fastest safe way to mitigate impact, such as rolling back a change, and the longer-term fix. Some agents can carry out approved actions and verify whether a mitigation worked.
  • Communication: Some agents also cover communication to stakeholders, such as providing status updates, handovers, and post-incident summaries.
  • Learning: Records relevant data points that helped resolve the issue, so that the next investigation has historical context, improving the performance of core capabilities.
  • Prevention: Some agents also help prevent issues by looking earlier in the software lifecycle, reviewing changes before they reach production or as changes go through a canary deployment.

Additional SRE agent capabilities include: checking releases, scheduled health checks, prevention recommendations, and improving observability.

1. Setup and configure the agent

Set the agent up for success

The agent understands your system through the services defined in your incident tooling. It works best when service mappings between your tooling ecosystem are consistent. For example, one service in your incident platform corresponds to one deployable unit, the same unit your observability tooling and repositories use. When the service catalog, observability tags, and repositories do not match, the agent may point teams to the wrong service.

SRE agents can typically pick up well-established relations and map scattered data sources. However, to maximize agent efficiency, you can also enrich the investigation triggers (e.g., a set of alerts) with information about the service, environment, region, cluster, and namespace to minimize investigation cold starts.

Which data sources does an SRE agent need?

An agent can reason only over the evidence it can reach. The table lists the data sources relevant to SRE agents, what the agent does with each, and the minimum setup needed.

Data source

What the agent uses it for

Minimum setup

  1. Alerts and incident management information (incidents, responders, etc.)

Investigation trigger, alert details, triage, analyzing past incidents, responder information, escalation policies and related incidents

Alerts routed to the right service, with enriched payloads and consistent urgency

  1. Telemetry (metrics, events, logs, and traces)

Query and validate issue symptoms, testing hypotheses against live behavior

Read access, log groups, and label filters for each service if applicable

  1. Change events (apps, configuration, infrastructure)

Linking deploys, configuration, and infrastructure changes to the issue

Deploy pipelines and repositories sending change events to the matching service

  1. Runbooks or instruction files

Following team and service procedures

One operational runbook linked to each team or service

  1. Post-incident reviews

What happened in past incidents, known fixes, semantic relations between symptoms, diagnosis, and mitigation

Read access to post-incident reviews

  1. Human input such as text, speech, notes, and other relevant artifacts

Following what responders find and decide during an incident

Routing of incident channel information to the agent, or the agent added to incident channels

  1. Source code

Localizing faults, reading recent changes, and working out service upstream and downstream dependencies

Read access to the repositories behind each service in scope

  1. Infrastructure and cloud APIs

Checking resource state, such as pods, instances, and quotas

Read-only access, scoped by account, region, or cluster

  1. Service catalog and dependencies (can be inferred from other sources)

Walking upstream to causes and downstream to impact

Each service mapped to its catalog component, with upstream and downstream dependencies declared

  1. Teams and organization structure

Collecting further business context about people

Who owns the services or who is on-call

 

The importance of each data source varies by incident type and its root cause. Although an agent can work with only alerts and an incident management platform, the more data sources you configure, the better the agent performs. If you have to prioritize a combination of 3 to 4 data sources, the following have proven the most relevant according to PagerDuty ablation studies: 

  • Traces: Traces carry the most information, but not all companies have distributed tracing or proper service coverage.
  • Logs, events, and metrics: Logs, events, and metrics are typically a great data source, contributing significantly to a more accurate diagnosis. 
  • Code and human input: Code and human input have also proven important in specific incidents, especially more complex ones. 

Record a baseline

Before the agent goes live, record current MTTR, mean time to acknowledge, time to identify the root-cause and the correct remediation, and how long responders spend gathering context, per team or service where you can. A baseline helps show the agent’s effect using data, not anecdotes.

Access and security threats

  • Least-privilege credentials: Create well-scoped credentials for each connector and tool the agent will use. 
  • Connector scope: Define what connectors can be used broadly and which ones have a restricted scope (e.g., per team).
  • Permission review: Review the agent’s permissions and ensure it can’t be used to access restricted content or elevate someone’s permissions. 
  • Action levels: Define what actions you want the agent to do autonomously, which require permission, and which are forbidden. 
  • Untrusted input: Logs, alerts, tickets, and chat are untrusted input. Text planted in them can redirect the agent or leak data (OWASP has published guidance on this). Defenses include egress allowlists, standing instructions that can’t override safety rules, and memory that outside text can’t poison.
  • Incident access: Ensure that people without access to an incident can’t use the agent to access information about it.
  • Approval governance: Define who may approve, track the approval rate, and avoid approval prompts people click through without thinking.

Mapping your distributed system 

The hardest incidents to diagnose are the ones where the service that alerts is not the service that failed. For example, a checkout API times out because a queue two hops away has stopped draining. By having a dependency graph, the agent can walk upstream to find candidate causes and downstream to identify the blast radius of the issue and then test both against runtime evidence.

However, creating a graph of your distributed system and keeping it up to date may be challenging. Each way of building that graph has trade-offs:

Source of the graph

Strengths

Weaknesses

Generated by a language model on request

Works on a small estate with no setup

Slow and costly to repeat, inconsistent between runs, and hard to check

Distributed tracing

Shows real calls with real timings

Covers only sampled, instrumented paths, often misses components like queues and databases, and is expensive

Service catalog or wiki

Captures human knowledge and ownership

Accurate on the day it is written, then drifts from the code

Static analysis of code, configuration, and infrastructure definitions

Versioned with the code and repeatable; each dependency can cite the file and line it came from

Has to be taught new frameworks and libraries

History of which services alert together

Shows which services fail together in practice

A clue, not proof of a dependency

Most teams combine several. Whichever you use, ask whether the agent can show why it believes two services are connected, and look for a graph that updates when the code changes.

Write runbooks/instructions the agent can use

Runbooks or instruction files written for people also help an agent, but a few changes make them more useful. The most valuable content is often the plainest: the exact data source and label names the agent needs for its first query.

Here are the recommended changes:

  • Use one entry point per service or per team. Keep it short and link to detailed pages rather than pasting everything in. The agent reads the whole entry point on every incident.
  • Put infrastructure context first. List the tech stack, log data source names, log groups, metric namespaces, key label filters (such as namespace, environment and service name), and links to dashboards, repositories and architecture documents.
  • Include sample queries and commands you have run. Examples are log and metric queries and the diagnostic commands from your standard procedures.
  • Organize procedures by problem. For each known failure mode, give the symptoms, the steps in order, what success looks like, and what to do when a step fails.
  • Link the runbook to the service in one place. A static link on the service is easier to maintain than a rule that injects it into every alert. 

A minimal runbook entry point:

# Service: payments-api

## Infrastructure

Tech stack: Go, Kubernetes, PostgreSQL, Kafka

Logs: data source “prod-logs-us-west-2”, labels namespace=payments, env=production

Metrics: “Payments overview” dashboard <link>

Repository: <link>    Architecture: <link>

## Known failure modes

### Database connection pool exhausted

Symptoms: p99 latency above 2 s, “pool timeout” errors in logs

Check: <log query that has been run and returns data>

Mitigate: scale replicas, or roll back the last deploy

Escalate: database on-call if pool usage stays above 90%

Customize how the agent behaves

Agent skills

While a runbook can influence how the agent operates when investigating a service, defining agent skills can help customize how the agent behaves. A skill is a written procedure the agent follows for a particular kind of problem: how your team investigates rate-limit errors, checks the blast radius, structures a status update, or validates a database failover. Skills turn what experienced responders know into behaviors the agent follows. Some items to consider when writing skills:

  • Write the trigger as carefully as the steps. The description determines when the agent uses the skill, so name the symptoms and the phrases responders use.
  • Take parameters from the incident. Write the skill to read the service, cluster, namespace, and time window from the alert instead of asking the responder. During critical moments, responders lose patience with an agent that needs a lot of input before it runs a query.
  • Run independent checks in parallel. Specify how to combine the results.
  • Define success and failure. Say what the output must contain and what to do when a data source is unavailable.
  • Scope skills by team. If your agent allows it, make each skill available only to the team that owns the procedure.

A skill example:

Name: analyse-rate-limit-errors

Description: Use when a service reports HTTP 429 or rate-limit errors from an upstream API.

Inputs (read from the incident): source service, upstream service, cluster, namespace,

  time window (incident start minus 30 minutes, to now)

Steps, run in parallel:

  Metrics: request count by response code, source to upstream. Report 429 volume,

    share of traffic, peak burst and trend.

  Logs: rate-limit errors in the namespace. Report affected accounts and first and last occurrence.

  Traces: failing requests, with latency and operation.

  Correlate: open incidents on the upstream service or provider.

Output: volume and share of traffic, bursty or sustained, affected accounts,

  likely provider, and whether it is likely to clear on its own.

If a step fails: say so, and continue with the others.

Different teams will want different agent behavior. A database team may want detailed technical analysis while a web team may want a short summary. Production calls for more caution than staging. If your agent supports multiple profiles, give each one its own instructions, response style, risk tolerance, and set of connections, and assign each profile through the routing that sends incidents to that team.

Agent autonomy

In many agents, you can configure the permissions associated with each tool as follows:

State

What the agent can do

Typical examples

On

Use the tool without asking

Search logs, read metrics, fetch files, read runbooks

Ask

Propose the action and wait for a human to approve it

Restart a pod, scale a deployment, roll back a release, run an automation job

Off

Cannot see or use the tool

Delete resources, change security groups, merge code

As you go through iterations of the SRE agent rollout, you can gradually increase the level of autonomy as follows:

  • Start read-only and widen autonomy gradually. Move an action from Ask to On only after the agent has proposed that action correctly many times and you have a quick way to undo it.
  • Give the agent its own identity. Attribute actions to the agent, not to whoever is on call, so your audit trail shows what the agent did.
  • Enforce limits. Filter the tools the agent can see, and also scope the credentials behind them, so a mistake in one is caught by the other.

Agent triggers

The agent should start investigating as early as possible, even before anyone is paged or acknowledges the issue. One option is to trigger the agent based on the routing or escalation rules for each team, rather than relying on responders to manually invoke it. This way, by the time a person starts to investigate, the agent has already started to triage and diagnose the issue. 

Other options include triggering the agent based on system anomalies or adopting a more preventative stance and using the agent to understand the consequences of changes to the system. Given how LLMs work, these less deterministic options may increase the chance of hallucinations due to the absence of clear impact evidence and are harder to optimize.

Validate the setup before you rely on it

After setting up a service, trigger a test incident on it and check that the agent:

  • Found and used the service’s runbook
  • Ran log and metric queries that returned data
  • Surfaced recent changes and dependent services
  • Posted results where the team expected

Replaying an old incident is a poor test of new configuration. The agent will be biased if the old incident (or parts of it) are loaded into context. Use exercises such as Failure Fridays and Chaos Engineering techniques to ensure the agent is working as expected. 

Agent memory

Several agents provide long-term memory, with some automatically populating it or proposing memory entries. However, distributed systems are subject to frequent changes. Thus, a memory can eventually become invalid, or contradictory facts can appear. Both can come from out-of-date knowledge bases or input collected during incidents. Some agents can find these contradictions and flag them to the user.

Choose where to start

Each organization has its own SRE agents rollout procedures. These suggestions may help with a more iterative rollout:

  • Start with well-instrumented services in the backend: Bring front-end and low-urgency services in later, once you know what good output looks like.
  • Aim for production services: Placeholder paging services, end-of-life services, and archived repositories inflate the target list and delay setup. 
  • Pick services with clear ownership: Services with clear ownership tend to have better monitoring, runbooks, and knowledge-bases, increasing the chances of success of the first iterations and keeping the focus on agent tuning.
  • Start with the services that have the highest toil, with issues that take longer to resolve: Rank services by incident volume and mean time to resolve (MTTR), and configure the highest-ranked services first. This makes it easier to see the impact and returns of the SRE agent.

2. Work with the agent during an incident

What to expect from an SRE agent’s conclusions

Treat the agent’s conclusions as hypotheses backed by evidence, not as verdicts. In controlled fault-injection tests across several agents, almost every agent cited evidence for its conclusions, yet the strongest averaged only about two-thirds on root-cause diagnosis, and all of them sometimes pointed responders at components that were not involved. Citing evidence and being right are different things, so responders should check the evidence before acting on it. 

These diagnosis numbers have been rising quickly as new and better LLMs come out. Additionally, several agents learn from past situations, so you can expect continuous improvement as context builds up. However, benchmarks show that the agent’s value is also uneven across types of issues, depending on the context provided and the system bias.

The first minutes

  1. Read the triage summary: Read the triage summary and potentially the diagnosis before opening dashboards. Check the impact estimate and the related incidents it lists.
  2. Check the top hypotheses: Look at the top hypotheses and the evidence behind each. Direct evidence observes the fault itself, such as a configuration change that removed a required setting or an error that names the failing dependency. Circumstantial evidence, such as an error spike that lines up in time, helps define where to look, eventually saving time.
  3. Guide cold starts: If confidence is low or something does not match what you expected, you can ask which alternatives the agent ruled out and guide the agent towards more promising investigation lines.
  4. Put mitigation first: If a recent change lines up with the start of the incident, rolling it back is often faster than understanding the bug. You can discuss the long-term fix and how to prevent it in the post-incident review.

Steer the investigation

To get the best results, treat the agent as a member of the response team.

  • Give it context it cannot see. This includes customer reports, what you have already ruled out, a maintenance window, or a third-party status page. 
  • Report the outcome of each step. When a suggested check or fix fails, say so. The agent can offer an alternative instead of repeating itself.
  • Redirect the agent. Step in when an explanation no longer fits the evidence. 
  • Keep the discussion where it can follow. Agents that read the incident channel and call transcripts have more context to help steer the investigation. Check confident claims from people as closely as you check the agent’s conclusions. When a teammate says, “I found it. It’s the database,” that is still a hypothesis. 

What to ask

Use specific prompts to get specific answers, or make the prompts below part of agent skills. These have proved useful while experimenting with different SRE agents:

Prompt

What you get

What are the likely root causes, and what evidence supports each?

Ranked hypotheses with the reasoning and the evidence to back it up

What changed on this service and its dependencies before the incident started?

Deploys, configuration, and infrastructure changes in that window

Have we seen this before? How was it resolved?

Similar past incidents and the steps that fixed them

Are related services affected?

Open incidents on upstream and downstream services, and the likely blast radius

Search the logs for this error between these times.

The query it ran, the results, and what they suggest

Should I do step X or step Y first?

A recommended order and the reasoning

What is the fastest safe way to stop the impact?

Mitigation options, with the risk of each

Draft a status update for stakeholders.

A plain-language summary of impact, progress, and the time of the next update

Generate a playbook from this incident.

The investigation and fix as reusable steps for next time

How can we stop this from happening again?

Follow-up actions for the service(s) and its infrastructure

What happened so far?

Catch up people who join in the middle of an incident

After the incident

When a problem is mitigated, several actions have proven to be useful for SRE agents:

  • Resolve: Resolve the issue in a way the agent registers the symptoms, the root cause, what was attempted, and how it was resolved. The more details, the better.
  • Update the playbook: Ask it to generate or update the playbook for this issue, and have someone who worked on it check it for accuracy. This will capture incident response best practices of your organization in a way that both agents and humans can leverage in the future.
  • Hand off follow-up work: Pass follow-up work to other agents. The agent’s analysis can seed the ticket for the fix, and a coding agent can draft the change for review.
  • Review the agent’s role:, Discuss what went well and what to improve during post-incident reviews. 

Help it learn

Most agents keep some memory between incidents such as:

  • Incident summaries: What happened, what was checked, and what did and did not work. Agents use these summaries to find similar past incidents.
  • Alert playbooks: The investigation steps, useful queries, and fixes for a recurring alert, updated each time the alert fires.
  • Service profiles: An evolving description of each service’s stack, dependencies, links, and known issues.

Memory makes the agent better only if it is accurate. Find out how to view, correct, and delete what your agent remembers, and make that part of routine maintenance. Systems also tend to change over time, so it is important to refresh the memory and remove lessons that turned out to be wrong. Some agents do this by using knowledge graphs with the capability of semantically connecting facts.

You can set a feedback cadence aligned with how you operate. Here are a few examples:

  • At handover or post-incident reviews: At the on-call handover or during a post-incident review, review where the agent helped, where it did not and why, and turn the gaps into instructions or skill changes.
  • Per incident: Mark responses as helpful or misleading, if the agent supports it.
  • On a regular cadence: Review memory and runbooks for the services with the most incidents, and check if the agent is producing the right outcomes (see section below).

3. Get more value from the agent

Where the value comes from

  • Coverage. The agent helps only on services it is configured for and incidents it is triggered on. Configure the services with the most toil first and trigger agent work automatically.
  • Richer evidence. Each source you add, whether runbooks, change events, dependencies, or transcripts, removes a step a responder would otherwise do by hand.
  • The first minutes. The largest savings come from gathering context and forming hypotheses before a person joins. In early pilots with different agents, log searching alone saved some responders up to 15 minutes of manual querying per incident.
  • Repeat incidents. Memory, playbooks, and past incident analysis help resolve the second occurrence of a problem faster than the first.
  • Model choice. In like-for-like tests, the more capable model diagnosed better in every comparison, at a higher cost. For investigation work, reasoning quality matters more than speed, which may come at the cost of increased misdirections. Some agents allow you to select the model or bring your own LLM provider.
  • Work around the incident. Status updates, handovers, post-incident drafts, and follow-up tickets take a large share of the team’s time. An agent can draft them quickly and work with other agents to automate these processes.

Measuring it

Measure outcomes, quality, and adoption together. Outcome metrics without quality metrics hide misleading answers, and adoption metrics without outcome metrics hide usage that saves nothing.

Category

Metric

How to measure

Outcome

Mean time to resolve 

Pilot incidents compared with a baseline of comparable incidents

Outcome

Mean time to mitigation action

From the alert firing to the correct fix being identified, by the agent or a person. Track it separately from MTTR, because carrying out the fix may be outside the agent’s control

Outcome

Time to first useful insight

Median time to the first output a responder rates as useful

Outcome

Toil saved

Responder estimates of time saved on gathering context, forming hypotheses, and communicating

Quality

Diagnosis and service localization accuracy

Share of incidents where the root cause, or at least the originating service, was right

Quality

Escalation rate

How many times a human has to intervene to achieve a successful outcome

Quality

Misleading recommendations

Share of suggestions responders mark as wrong or misleading

Adoption

Engagement

Share of incidents where responders interacted with the agent, and repeat use by the same people

Adoption

Readiness

Share of in-scope services that meet the readiness checklist

Cost

Cost and latency per investigation

Usage charges and time to complete, tracked from the first day of the pilot

 

Report against the baseline on a regular basis. You can pair the numbers with more qualitative insights such as situations where the agent changed what responders did and one where it did not.

4. Evaluate and compare agents

Why evaluation is hard

Comparing SRE agents fairly is harder than it looks. Replaying past incidents is the obvious approach, but has its limits:

  • Post-incident reviews are written for people and are often incomplete.
  • Hindsight leaks into the replay, and even the incident title can reveal the answer.
  • Replays cover only the incidents you have already experienced.

Third-party agents query live systems through their own integrations, so they cannot be wound back to the moment an incident started. Public benchmarks mostly run on demo applications that language models have seen in training, and few include source code as evidence. 

Design the trial

  1. Agree on use cases and success criteria: Pick a few use cases that matter to the organization.
  2. Prepare and test the setup and environment Make sure that integrations are connected and the agent identity has permissions, ensure connectivity to the data sources listed in section 1 such as observability, code, collaboration tools, etc. If applicable, configure skills and set up the necessary prompts to adjust behavior
  3. Replay recent incidents: Replay one or two recent high-severity incidents whose telemetry and context still exist. Confirm what data is available, run the agent without showing it the human root-cause analysis, then compare its output with the documented analysis and timeline.
  4. Run a live pilot: Spend at least a couple of weeks with two teams whose senior engineers will give structured feedback. Run incidents end to end and compare the agent’s conclusions with the engineers’ judgment.
  5. Inject faults where you can: Use a development or staging environment where you are allowed to break things because it gives you exact ground truth. Incidents that use real logs and runbooks are more telling than fully synthetic ones.
  6. Test learning: Create two similar incidents on the same service a few days apart, and check that the second investigation uses what was learned in the first.
  7. Use the same conditions for every agent: Give every agent the same incidents, data access, and time window, with the same neutral prompts.
  8. Score missing capabilities as missing: If an agent does not offer something on the scorecard, give it a low score rather than leaving the row out.

Score on several axes

A root cause is rarely a single fact, and the fastest mitigation may not fix the root cause in place. Score each agent on several axes:

Axis

Question

Issue localization

Did it identify the service where the fault originated, not just where it showed up?

Root-cause match

Did it identify the root cause? Give partial credit for partial answers.

Evidence grounding

Is each conclusion backed by specific logs, metrics, code, or changes?

Explanation

Could a responder follow the reasoning, including the alternatives it ruled out?

Mitigation

Did it recommend the fastest safe way to stop the impact?

Misdirection

Did it point responders at components that were not involved? This catches agents that list every possible cause without committing to one.

Speed

How long until the first useful output, and until the correct diagnosis?

Cost

What did each agent run cost?

If you use language models to help with scoring, use judges from more than one model provider and check a sample against human scores.

Collect responder feedback

After each significant incident, ask pilot participants for feedback:

  • Did the agent’s output change what you did in the first 15 minutes? How?
  • What did you trust, and what did you ignore? Why?
  • What context was missing?
  • Did it reduce toil or add work?
  • Did you feel able to act, or did you still need to escalate?
  • Would you want it in your next incident?

This will help inform how to fine-tune the SRE agent.

Capability checklist

Use these capabilities when comparing agents or planning what to configure or build. Each item comes from a problem agents run into during real incidents.

Capability

Why it matters

Starts on its own given a trigger

Triage is ready before a person joins

Runs independently of your infrastructure

It must stay available during your service disruption

Keeps investigating for the whole incident

Evidence arrives over time, and early signals are often inconclusive

Accepts steering during the investigation

Responders know things the telemetry does not

Links each hypothesis to evidence, separating direct from circumstantial

Responders can check the reasoning quickly

Records the alternatives it ruled out

Responders see what was excluded as well as what was concluded

Detects cascading failures across systems and layers 

Faults often originate away from the alerting service

Correlates recent changes

Many incidents follow a deploy or configuration change

Interprets live chat and key moments

Human context improves diagnosis

Remembers past incidents, with a way to view, correct, and delete memory

Learning helps only if it is accurate

Supports defining procedures and configurations with different scope

Teams investigate differently and have different levels of access

Sets permissions per tool, with approval steps, under its own identity

Autonomy can grow safely and stays auditable

Works across different environments

Responders will use the tools they are used to working with

Keeps its underlying models current

Model quality drives diagnosis quality

Frequently asked questions

How do you set up governance and approval workflows for an SRE agent?

Define what actions you want the agent to do autonomously, which ones require permission, and which ones are forbidden. Define who may approve, track the approval rate, and avoid approval prompts people click through without thinking. Give the agent its own identity, so your audit trail shows what the agent did.

What do you need before you roll out an SRE agent?

Start with well-instrumented production services that have clear ownership and the most toil. Connect alerts and incident data, telemetry, change events, runbooks, and source code. Record your current MTTR and other response times before the agent goes live, then run a live pilot of at least a couple of weeks with two teams.

How should you scope access and permissions for an SRE agent?

Create least-privilege credentials for each connector and tool the agent uses, and restrict some connectors to specific teams. Make sure people without access to an incident can’t use the agent to see information about it. Filter the tools the agent can see and also scope the credentials behind them, so a mistake in one is caught by the other.

What are the autonomy levels for an SRE agent?

Many agents let you set each tool to On (the agent uses it without asking), Ask (the agent proposes an action and waits for a human to approve it), or Off (the agent can’t see or use the tool). Start read-only and move an action from Ask to On only after the agent has proposed that action correctly many times and you have a quick way to undo it.

How do you evaluate an SRE agent’s accuracy?

Score each agent on issue localization, root-cause match (with partial credit), evidence grounding, explanation, mitigation, misdirection, speed, and cost. Give every agent the same incidents, data access, and time window, with the same neutral prompts. If language models help with scoring, use judges from more than one model provider and check a sample against human scores.

How do you measure the return from an SRE agent?

Measure outcomes, quality, and adoption together against a baseline recorded before launch. Track MTTR, time to mitigation action, time to first useful insight, toil saved, diagnosis accuracy, misleading recommendations, engagement, and cost per investigation.