Moving from firefighting to architecture: Reimagining operations with autonomous agents

The main point of this discussion is that current SRE workflows do not fit well with today's general-purpose agent frameworks. Coding agents are designed to generate text in controlled settings, but SRE agents must function within the unpredictable, high-stakes environment of production. By moving from imperative scripting to a declarative, Kubernetes-style model, teams can stop acting as firefighters. The result is not just faster troubleshooting, but a shift in the SRE role from reactive maintenance to proactive reliability architecture. Engineering leaders who adopt this approach now gain a competitive edge: they stop managing individual incidents and start managing the systemic patterns that prevent them.

The hidden costs of general-purpose frameworks

Most teams trying to automate production operations start by wrapping general-purpose agent frameworks in simple prompts. Andre Elizondo, Head of Product at Mezmo, calls this a day one trap. These frameworks are unopinionated, which forces teams to manually build the reasoning loops, retry logic, and safety guardrails that production systems require.

"It's like the building of agents especially for managing production is very much like the iceberg analogy... Everybody's thinking about the tip of the iceberg and hey how do I just take a model, wrap it in a simple agent framework and give it a prompt? And in reality there is a lot of things that are needed to actually make that reliable."

-- Andre Elizondo

When teams ignore the bottom of the iceberg, they build brittle systems that fail without warning. This leads to context bloat, where agents ingest weeks of traces, causing them to hallucinate or exceed their context windows. The systems-thinking approach is to recognize that production operations require context engineering--deliberately restricting and shaping data for the agent--rather than feeding it raw telemetry.

The paradox of declarative autonomy

The most non-obvious insight is that increasing agent autonomy requires more structure, not less. By adopting a declarative model, where SREs define the desired state rather than the imperative steps, teams can enforce consistency across agent behavior. This mirrors the transition from manual sysadmin work to Kubernetes-driven infrastructure.

This approach creates a graduation path for autonomy. Teams do not need to choose between a human-in-the-loop and a fully autonomous system. They can configure agents to act as co-pilots and gradually grant more authority as the system proves its reliability. This creates a feedback loop: as the agent succeeds, it gains trust, which allows it to handle more complex remediation and frees human engineers to focus on architectural design instead of incident response.

Where immediate pain creates lasting moats

The most significant competitive advantage comes from investing in observability for the agents themselves. Most teams treat agents as black boxes. By instrumenting agents with OpenTelemetry, engineers can trace the reasoning of an agent just as they trace a service request.

"There is an aspect of how you provide the ability for somebody to say hey, these types of tools which maybe are read only or sorry, write... I want to make sure that I provide a very hard kind of barrier, like for the agent to be able to know that that is something that's insanely sensitive and it needs to ensure that it asks for help before continuing on."

-- Andre Elizondo

This transparency is the moat. While competitors struggle to debug opaque agent failures, teams that have invested in agent-legible observability can identify exactly why a model chose a specific path. This turns the agent from a source of anxiety into a reliable, auditable component of the production stack.

Key action items

• Audit your current agent framework: Evaluate if your setup requires imperative scripting for every step. If so, plan a transition to a declarative configuration over the next quarter to reduce maintenance overhead.
• Implement scratchpad patterns: Stop feeding raw telemetry directly into the context window. Build or adopt a layer where agents interact with tool outputs via file-based storage to preserve context and reduce token waste. (Immediate)
• Instrument with OpenTelemetry: Ensure every agent action produces traces that mirror your production service telemetry. This pays off in 6 to 12 months by turning black box agent errors into debuggable data.
• Define your one-way door policies: Explicitly map which remediation actions require human approval. Configure your agent framework to enforce these barriers at the tool level, not just the prompt level. (Immediate)
• Close the institutional knowledge loop: Instead of maintaining static runbooks, build a feedback mechanism where agents can propose updates to documentation via PRs after an investigation. This creates a compounding knowledge base over 12 to 18 months.
• Establish a graduation framework: Define the threshold of successful runs required before an agent is granted more autonomy. Treating this as a rigorous testing process, rather than a binary toggle, mitigates long-term operational risk.