The Hidden Costs of AI's "Always On" Future

The rapid advancement of AI, particularly in agentic systems, is pushing users and developers to the brink of what's cognitively sustainable. This conversation reveals a critical, often overlooked consequence: the potential for "AI psychosis" and burnout due to the relentless, always-on nature of these tools. While the immediate benefits of AI agents seem boundless, the hidden costs lie in their impact on human cognition, the erosion of traditional work structures, and the ethical quagmire of AI-driven marketing. This analysis is crucial for anyone building, deploying, or simply using AI, offering a framework to navigate the complex downstream effects and build more sustainable, human-centric AI integrations. Ignoring these implications risks not only individual well-being but also the long-term viability and ethical integrity of AI adoption.

The Unseen Toll: Cognitive Load and the "Always On" Agent

The allure of AI agents that never sleep, that can "bang on the Anthropic access routes continuously," as Beth Lyons notes, presents a double-edged sword. While the immediate benefit is increased productivity and the ability to offload tasks, the underlying consequence is a significant cognitive load on the human orchestrator. Andre Karpathy's admission of being in a state of "AI psychosis" since December, managing swarms of agents that write 100% of his code, highlights the unsustainable nature of this model. The sheer mental effort of managing these autonomous systems, even when they are performing the heavy lifting, can be exhausting.

This isn't just about developers pushing their limits; it's a broader societal shift. Simon Willison, interviewed on Lenny's podcast, articulates a fundamental truth: "There's a limit on human cognition in how much you can hold in your head at one time, and it's very easy to pop that stack." This limitation is precisely what AI agents, in their tireless efficiency, can exacerbate. When AI agents become extensions of our work, blurring the lines between human and machine effort, the traditional cues for rest and recovery--the end of a workday, the feeling of completion--can become obscured. The throttling of third-party access to models like Claude by Anthropic, forcing users onto API pricing or credits, is a direct response to this unsustainable demand. It's a forced "rate-limiting" for the AI infrastructure itself, a signal that the current model of infinite, free access to powerful AI agents is not viable.

The conversation then pivots to the idea of creating external backup systems, not just to avoid API rate limits, but to reintroduce human-imposed boundaries. The analogy of pets or buddies serving as natural cues for rest is telling. It suggests a need for external mechanisms, perhaps even built into our AI interactions, to signal when enough is enough. The move towards local, open-source models like Google's Gemma 4, which can run on personal devices, offers a potential off-ramp. This decentralization allows for continuous use without hitting external API limits, but it also shifts the burden of managing resources and potential burnout back onto the individual user. The challenge, then, isn't just about accessing AI, but about managing our relationship with it to prevent a collapse of our own cognitive capacities.

"There's a limit on human cognition in how much you can hold in your head at one time, and it's very easy to pop that stack."

-- Simon Willison

The Corporate Knowledge Paradox: Skill Capture vs. Abstraction

The viral spread of the "colleague.skill" repository in China, coupled with discussions around documenting personal work product, touches upon a profound fear: the obsolescence of individual expertise in the face of AI. The anecdotal story of companies letting go of employees after their knowledge has been codified into AI-usable formats highlights a stark, immediate consequence. This creates a perverse incentive structure where documenting your own value could, paradoxically, make you redundant. The "distiller" repos, designed to strip personal decision-making from documented processes, represent a defensive maneuver by individuals seeking to retain their unique value.

However, Andy Halliday offers a compelling counter-perspective, reframing the issue from individual skill capture to corporate knowledge abstraction. He points to the rise of enterprise-focused AI offerings that aggregate data from various systems (Slack, CRM, etc.) to create a unified, AI-comprehensible view of corporate knowledge. This approach moves beyond individual expertise to focus on the underlying concepts, sequences, and processes that constitute organizational value. The implication is that the future of work may not be about individuals guarding their unique skill sets, but about abstracting that knowledge into systems that AI can leverage.

This shift has significant downstream effects. If corporate knowledge becomes an abstracted, AI-accessible entity, the role of the individual human in an organization fundamentally changes. While near-term human intervention will likely remain necessary, the long-term trajectory suggests a move towards AI-native startups that can offer services at a lower cost and higher automation. This, in turn, puts pressure on larger organizations to adapt or risk becoming obsolete. The fear of job displacement, prevalent in Western discussions around robotics, is juxtaposed with the Eastern perspective, where AI and robotics are seen as essential for industrial survival and addressing labor shortages. The consequence of this divergence in perspective is a potential widening of the competitive landscape, where nations and companies that embrace AI as a tool for systemic efficiency will outpace those that view it primarily as a threat to individual roles.

"When we speak of corporate knowledge, we mean in the human organization that's retaining the expertise that's been built over decades in an organization."

-- Andy Halliday

MedVee: The AI-Powered Mirage and the Scale of Deception

The story of MedVee, initially lauded as a groundbreaking $1.8 billion solo-founder AI company, serves as a cautionary tale of AI-enabled fraud. Gary Marcus's aggregation of critiques reveals that the company's success was not built on innovative AI technology but on an "affiliate marketing network" leveraging "800 AI-generated fake doctors on social media." This elaborate scheme employed falsified headers, spoofed domains, and deceptive subject lines, all amplified by AI-generated deepfakes for marketing. The downstream consequences are severe: a class-action lawsuit, an FDA warning for misbranding violations, and the erosion of trust in AI-driven business models.

The critical insight here is not just the existence of the fraud, but the scale at which it was perpetrated, directly enabled by AI. The ability to generate convincing fake personas, manipulate marketing channels at scale, and create a veneer of legitimacy for a fraudulent operation is a chilling demonstration of AI's potential for misuse. The narrative suggests that the very tools that enable rapid innovation and efficiency can also be weaponized for massive deception. The MedVee case is a canary in the coal mine, signaling the advent of billion-dollar fraud schemes that exploit consumer behavior through sophisticated AI manipulation