Building Agentic AI Infrastructure Beyond Chatbots

The emergent infrastructure for agentic AI is not just about smarter chatbots; it's about fundamentally re-architecting how organizations interact with technology, revealing hidden complexities in identity, control, and operationalization that most overlook. This conversation with Craig McLuckie, CEO of Stacklok, illuminates how the principles honed in the cloud-native era, particularly around Kubernetes and declarative infrastructure, are crucial for building secure, scalable, and manageable AI-native applications. Those who grasp these underlying systems dynamics--moving beyond simple tool invocation to robust agent orchestration and secure access--will gain a significant advantage in deploying AI not as a novelty, but as a core operational capability. This is essential reading for engineering leaders, platform architects, and anyone tasked with integrating AI into enterprise workflows beyond basic chatbot interactions.

The Yellow Brick Road to Agentic AI: Beyond the Emerald City

The excitement around generative AI and large language models (LLMs) often focuses on the dazzling capabilities--the ability to write code, generate text, or answer complex questions. This is the "Emerald City" of AI, a vision of a future where intelligent agents act as coworkers. However, Craig McLuckie, CEO of Stacklok, argues that the real challenge, and the source of significant competitive advantage, lies in building the "Yellow Brick Road" to get there. This involves creating the robust, secure, and manageable infrastructure that underpins these powerful, yet often unpredictable, agentic systems.

McLuckie draws a powerful parallel to the early days of Docker and Kubernetes. Docker solved the immediate developer problem of packaging applications, while Kubernetes provided the orchestration layer for complex, cloud-native deployments. Similarly, the Model Context Protocol (MCP) is emerging as a critical protocol that not only formalizes how LLMs interact with external systems but also provides the necessary guardrails for enterprise adoption. The non-obvious implication is that MCP isn't just an API wrapper; it's a fundamental architectural component for bridging the gap between the "stochastic" nature of AI agents and the deterministic requirements of enterprise systems.

The immediate benefit of LLMs is their natural language interface and ability to extract semantic meaning. However, asking them to directly interact with traditional APIs, handle authentication, or operate within strict security boundaries is "fiddly." MCP addresses this by describing the outside world in simplified natural language terms, backed by JSON schemas, allowing LLMs to reason about and invoke tools deterministically. This creates a "selectively permeable membrane" around an organization's existing systems, enabling controlled value flow in both directions.

This capability is transformative for workflows like that of a recruiter. Instead of manually jumping between email, LinkedIn, and a CRM, an AI assistant, empowered by MCP, can access these systems. MCP allows these disparate systems to be described with clear nouns and verbs--what a "candidate" is, what actions like "schedule interview" entail--formalizing them as discrete resources and tools for the AI. This formalization is key because it allows for granular control over authentication and authorization, ensuring that an agent acting on behalf of a recruiter can only access what that recruiter is permitted to access, and no more.

"MCP really represents this small, sharp protocol that you can use to start reconciling the behavior of systems that are accessing the real world and also setting up guardrails and controls."

-- Craig McLuckie

The challenge, as McLuckie points out, is that many organizations are currently tethered to vertically integrated AI platforms. While these offer easy entry points, they often lack the flexibility to integrate with the broader ecosystem of enterprise tools. The "developer's desktop" becomes the aggregation point, which is unsustainable and insecure for broader enterprise adoption. The real value emerges when this capability is decoupled from specific vendor solutions and made available through a unified platform. This requires building out an "MCP platform" comprising several key components: a secure runtime environment, a registry for discovering and vetting services, a gateway for unified access, and a control plane for managing policies and user mappings. Complementing this is an LLM gateway, enabling organizations to direct traffic to various models and institute tracking and policy management. These two gateways--MCP and LLM--form the essential "bookends" of any robust agentic platform.

The Hidden Cost of "Easy" Integrations

The allure of native integrations offered by AI providers is strong. They promise a quick path to functionality, like connecting an LLM to Slack or Google Workspace. However, this approach quickly hits a wall when organizations need to integrate with their unique, internal systems or require more nuanced control. McLuckie highlights that relying solely on these native integrations creates a fragmented experience and bypasses critical enterprise requirements for security, governance, and observability.

The "obvious solution" of having an agent directly invoke APIs using API keys stored as environment variables, for instance, is a critical failure point. This approach obliterates the user's identity and authorization context. An agent making a call with a generic API key cannot distinguish between users or enforce granular permissions. This leads to a situation where an agent might have unfettered access to sensitive data, posing a significant security risk.

"The starting point usually is, 'Okay, well, Claude will offer its own kind of native integration systems.' ... But at some point, you're going to start asking questions like, 'Well, what about all the other systems that I used to do my work? How do I start to expose those?'"

-- Craig McLuckie

The true downstream consequence here is the inability to scale AI safely. Without a structured approach to identity and authorization, organizations cannot confidently deploy agents beyond simple, isolated tasks. The "easy" integration today creates a significant technical debt and security vulnerability for tomorrow. The path forward involves robust token exchange mechanisms, where user credentials are exchanged for scoped tokens that grant agents only the necessary permissions. This requires a dedicated platform team to manage these complexities, but the payoff is the ability to securely expose a vast array of enterprise systems to AI agents.

The Proxy Paradox: Visibility and Optimization

When building bridges between AI and enterprise systems, the question arises: should this integration happen at the application layer or through a dedicated proxy layer? McLuckie argues strongly for the latter, emphasizing the critical benefits of visibility, governance, and optimization that a proxy provides.

Attempting to manage MCP connections directly within each application leads to a distributed, unmanageable mess. Debugging complex workflows involving multiple systems becomes a nightmare. A proxy layer, however, acts as a central point for observability. It allows tracing requests across different systems, identifying bottlenecks, and understanding how agents are interacting with tools. This is crucial for maintaining system health and diagnosing issues.

Furthermore, a proxy layer is essential for optimizing token usage, a significant cost factor in LLM interactions. MCP servers often describe numerous tools and resources, which, when exposed directly, can flood the LLM's context window with irrelevant information. This "tool pollution" can lead to increased token consumption and degraded performance, especially with smaller models. A proxy can intelligently amalgamate these tools, presenting a more refined and contextually relevant set of options to the LLM. This optimization can reduce input token consumption by 80-90%, making agentic systems more cost-effective and reliable.

"It reduces input token consumption by 80 to 90% when you have these things. And so that's a very big deal versus just allowing the models to access the tool selection..."

-- Craig McLuckie

Finally, a proxy enables fine-grained, user- or project-specific views of tools. The meaning of a "feature," for example, can differ drastically between a GIS system and a GitHub repository. A proxy can disambiguate these terms, presenting tools with semantically relevant descriptions tailored to the specific task or user cohort. This abstraction layer is not just about convenience; it's about ensuring that AI agents can effectively and reliably interact with the complex, nuanced world of enterprise data and functionality.

The Kubernetes Parallel: Declarative Infrastructure for Agents

The success of Kubernetes in managing complex cloud-native applications stemmed from its declarative nature--defining the desired state and letting the system reconcile it. McLuckie sees a similar future for agentic AI infrastructure. The goal is to move towards "reconciliation-driven infrastructure" where users describe their desired outcomes, and the system, potentially leveraging stochastic AI, works to achieve and maintain that state.

This could manifest as self-annealing, self-healing, and self-optimizing systems. Imagine describing a desired application state in natural language, having an AI generate the necessary Kubernetes manifests, and then having intelligent agents monitor and maintain that state. When deviations occur, these agents could attempt to reconcile the system, or if the problem is beyond deterministic solutions, invoke stochastic systems to diagnose and fix issues.

The packaging and deployment of agents themselves will likely adopt Kubernetes-like patterns. This involves defining agents as OCI entities and developing agent-specific platform systems that can be integrated into control loops and described as tools. However, a significant challenge remains: tracking agent behavior and establishing effective reconciliation loops for bounding agent actions. While evaluation metrics and human-in-the-loop systems offer partial solutions, the community is still grappling with how to build robust governance and control mechanisms for autonomous agents.

Actionable Takeaways

• Embrace MCP as a Foundational Protocol: Recognize that MCP is more than an API wrapper; it's a critical architectural component for secure and scalable AI integration. Invest in understanding and implementing MCP-based solutions.
• Prioritize Identity and Authorization: Do not rely on basic API key management for agent access. Implement robust token exchange mechanisms and leverage existing OIDC systems to de-scope claims and enforce granular permissions.
• Build or Adopt a Proxy Layer: For any significant AI integration, a dedicated proxy or gateway is essential for visibility, governance, and optimization of token usage. This is crucial for debugging and cost management.
• Leverage Cloud-Native Principles: Apply learnings from Kubernetes and the cloud-native ecosystem to agent infrastructure. Think in terms of declarative configurations, secure runtimes, and registries for managing AI services.
• Develop Platform Teams for AI: Treat AI integration as a platform engineering challenge. Dedicated teams are needed to build and maintain the infrastructure, security, and governance layers that enable safe and effective agent deployment.
• Focus on Developer Productivity Through Agentic Concurrency: As seen in development teams, orchestrating multiple specialized agents can dramatically boost productivity. Explore how to enable similar "superpowers" for knowledge workers by providing them with controlled access to these capabilities.
• Plan for Long-Term Agent Management: The challenge of tracking and bounding agent behavior is significant. Begin thinking about strategies for monitoring, evaluation, and control as you scale your AI deployments. This is a long-term investment that will pay off as agents become more autonomous.