Humans Managing AI Agents: The New Productivity Paradigm

The future of work isn't about humans doing more, but about humans managing more. Notion's co-founder, Simon Last, reveals a profound shift: the transition from a tool where individuals perform tasks to a platform where humans orchestrate a swarm of AI agents. This conversation uncovers the hidden complexities of building truly autonomous agents, from the intricate challenge of indexing vast, disparate data to the necessity of constant iteration in AI development. Those who grasp this paradigm shift early will gain a significant advantage, not just in efficiency, but in shaping the very nature of productivity. This is essential reading for anyone building or using productivity tools, offering a glimpse into a future where human oversight is the primary skill.

The Evolving Harness: Why Constant Rewrites Are the Key to AI's Promise

The initial excitement around AI, particularly large language models like GPT-4, was palpable. For Notion, this wasn't just about adding a feature; it was about reimagining the core of their product. Simon Last describes the immediate realization that AI represented a fundamental shift, leading to both short-term goals--like an AI writing assistant--and a long-term vision of a general-purpose AI assistant capable of complex, multi-step tasks within Notion. However, the journey from concept to reality has been far from linear. The "obvious" solutions, like simple text rewriting, were achievable quickly. But the more ambitious goal of creating agents that could autonomously build integrations, query databases, and weave information together required a much longer, more iterative approach.

This iterative nature is most starkly illustrated by Notion's internal "AI harness," the system that interfaces with AI models. Last reveals that this harness is rebuilt approximately every six months. This isn't a sign of failure, but a testament to the rapid pace of AI development and the need to design product infrastructure deeply around the current state of the technology.

"The time to rewrite has kind of been been decreasing just because i think like like progress has been accelerating. I think this is honestly a a really key thing and something that a lot of companies get wrong is just like doing one thing and then just like like sticking with it. You really do have to be keenly aware of what the current state of the model and the technology is and then designing the harness and system in the product deeply around that."

This constant rebuilding highlights a critical, non-obvious implication: the perceived "stability" of AI tools is an illusion. True mastery requires embracing flux and understanding that what works today may need a complete overhaul in six months. Companies that resist this constant evolution will inevitably fall behind, their AI capabilities becoming outdated before they've even fully materialized. This presents a significant advantage to those who can adapt quickly, turning the churn of AI development into a source of competitive separation.

Beyond the Monolith: The Unforeseen Complexity of Unified Data

One of the most significant technical hurdles Notion has tackled is semantic indexing and retrieval across disparate data sources. The initial success of Notion's internal AI writer, which operated on text within documents, was relatively straightforward. The real challenge emerged when they aimed to provide answers grounded in a user's entire workspace, including data from Slack, Google Drive, and other integrations. This venture into "Switzerland for models," as Last puts it, revealed that even established products often struggle with effective data indexing and retrieval.

The complexity lies not just in connecting these services, but in understanding the nuanced nature of each data type. Querying Slack messages is fundamentally different from querying Google Drive documents. Last emphasizes that a one-size-fits-all approach is insufficient. Achieving robust retrieval requires a deep understanding of "craft and love"--an empirical, iterative process of trying various queries, constantly rethinking retrieval mechanisms, and tuning the pipeline.

"Each each data source is a little bit special like you know you can't just apply a one size fits all to like querying slack versus querying google drive let's say they're they're completely different kinds of information and we found that there's just a little bit of like like craft and love that's goes into it in terms of like actually trying a bunch of different queries actually using it every day and constantly iterating and rethinking and and tuning how the retrieval works."

This insight has a cascading consequence: the perceived "solved" problem of data integration by other platforms is often superficial. Notion's commitment to deeply solving this for their users, even when it means tackling challenges others haven't, creates a durable advantage. The non-obvious implication is that the true value of AI in productivity isn't just in the model's intelligence, but in its ability to access and synthesize information from a user's entire digital life. This requires a level of engineering depth that most companies are unwilling or unable to invest in, leading to a significant competitive moat for Notion.

The Agentic Engineer: Amplifying Impact and Widening the Gap

The advent of coding agents has fundamentally altered how engineering and product development operate at Notion. Last describes a shift from simple tab completion to agents that can implement, verify, and maintain code. This has dramatically increased the ambition of what can be built and, crucially, the robustness of the resulting systems. However, this power comes with a caveat: it requires significant thought in architecting systems and establishing verification loops.

The most profound impact, according to Last, is on individual impact. The ability of engineers to deliver output has skyrocketed, making their effectiveness increasingly dependent on their willingness and ability to leverage these AI tools. This creates a widening chasm between engineers.

"I think the the gap is bigger. You can be like a hundred or thousand x engineer if you're using the tools right now. I think i think the the gap is much bigger. The the minimum bar has not changed but the maximum bar has has extremely increased."

This dynamic challenges conventional wisdom about team structures and individual performance. While small, agile teams remain desirable, the definition of a "10x engineer" is being redefined. It's no longer solely about raw coding skill, but about the ability to effectively manage and direct AI agents. This requires a new kind of "agent manager" rather than just a coder. The consequence of this shift is a more dynamic, albeit potentially more chaotic, internal environment with a proliferation of prototypes. The advantage lies with individuals and teams who embrace this new paradigm, understanding that their future productivity and impact will be measured by their mastery of agent orchestration.

From Tool for Work to Tool for Agents: The Fundamental Reorientation

Last articulates a core shift in Notion's foundational goal: "Before AI, our goal was to create the best tool for humans to directly perform their work. And then now, the goal is to create the best tool for humans to manage agents to do the work for them." This is not a minor adjustment; it's a fundamental reorientation of the product's purpose.

Interestingly, this shift doesn't negate the value of Notion's existing primitives. Documents, for instance, remain highly relevant as agents "love to write markdown documents." Similarly, databases are still essential for structured data, providing the necessary coordination and oversight for a "swarm of background coding agents" that would be unmanageable through chat threads alone. The implication is that the underlying needs of productivity--organization, communication, data management--remain, but the method of fulfilling those needs is evolving dramatically.

The advantage for early adopters of this agent-centric model is immense. They are not merely optimizing existing workflows; they are building entirely new ones based on human-agent collaboration. This requires a different kind of thinking, moving from direct execution to strategic delegation and oversight. The "messy and chaotic" nature of this transition, with more prototypes and experimentation, is precisely where innovation and competitive advantage are born. Those who can navigate this complexity and build effective agent management systems will define the next era of productivity.

• Embrace the Iterative Cycle: Recognize that AI development, particularly for complex agents, requires continuous iteration. Rebuilding internal systems like AI harnesses every six months is not a sign of instability, but a necessity for staying at the cutting edge.
• Deep Dive into Data Integration: Understand that true AI utility hinges on seamless, intelligent access to diverse data sources. Invest in robust semantic indexing and retrieval beyond superficial integrations.
• Cultivate Agent Management Skills: Focus on developing the ability to architect tasks, prompt effectively, and oversee AI agents. This is the new frontier of individual and team productivity.
• Prototype and Experiment Relentlessly: Foster an environment where building prototypes and experimenting with agent workflows is encouraged, even if it leads to initial chaos. This is where the most valuable insights and innovations will emerge.
• Invest in Agent Workflows: For personal use, dedicate time to building and refining custom agents for tasks like email triage or feedback routing. This immediate pain of setup yields significant long-term payoffs.
• Develop Internal Agent Literacy: Conduct workshops and hackathons to equip non-technical teams with the intuition and skills to leverage coding and custom agents for their workflows.
• Shift Your Mental Model: Reframe productivity not as doing work yourself, but as managing the agents that do the work for you. This fundamental shift in perspective is key to unlocking future efficiency gains.