The AI Feedback Loop: Why Your Intuition Is Failing You

The current debate about AI, which swings between fear of the future and dismissive skepticism, misses the most significant consequence: we are no longer just building tools. We are building an autonomous, persuasive layer that sits between humans and the truth. This creates a feedback loop where the system output becomes the input for future versions, turning the internet into a closed circuit of machine-generated consensus. For leaders and practitioners, the advantage lies not in predicting whether AI will go rogue, but in mastering the design stance. This means moving past the human-like veneer of chatbots to understand the brittle, incentive-driven architecture that governs their behavior. Those who treat AI as a partner will be manipulated; those who treat it as a system to be debugged will retain their agency.

The Hidden Cost of Fast Persuasion

The most dangerous aspect of AI is not its intelligence, but its velocity. Research suggests that AI models outperform world-class human debaters not because they are more logical, but because they use a Gish gallop, which is a barrage of facts delivered at superhuman speeds. This creates an immediate, visceral sense of authority that bypasses critical analysis.

"It really seems to be how many facts... how much evidence this AI can present in a given amount of time. And in the Hackenberg study... he limited the AI to the human speed of an interaction and found that it erases the AI's persuasive edge."

-- Kai Kupfertschmidt

When we look ahead, the consequence is clear: the persuasive edge of AI is a function of throughput. In a world where AI-generated content competes for attention, the system that can overwhelm the user cognitive capacity first wins the narrative. This creates a lasting disadvantage for human communication, which remains tethered to the slower, more deliberative pace of traditional discourse.

The Feedback Loop: When Bots Train Bots

We are seeing the emergence of Generative Engine Optimization (GEO), where the primary goal of content creation is no longer to inform humans, but to manipulate the crawlers that feed LLMs. This is a systems-level failure. As companies optimize their digital presence to be recommended by chatbots, they pollute the data pool these models use to learn.

The downstream effect is a flattening of the internet. We are moving from a diverse, discovery-based web to a frictionless, homogenized experience where the chatbot provides the answer, effectively killing the high-effort process of cross-referencing multiple sources. The system responds to these incentives by becoming a mirror of the most aggressive marketers, not the most accurate sources.

The Design Stance as a Competitive Moat

The debate over whether AI is alive or rogue is a distraction. As philosopher Daniel Dennett noted, we choose our stance toward a system. The intentional stance, which means treating the AI as if it has beliefs and goals, is what makes us vulnerable to manipulation. The design stance, which means viewing the AI as a series of circuits, instructions, and training incentives, is where the power lies.

"If your feeling is there's something wrong in the inner functioning of this thing, you start to get more interested in like, how was it built? What is it actually doing? How does it come to these conclusions? Can I trust it to work on its own? That's a different stance."

-- Joshua Rothman

By shifting to the design stance, practitioners can identify where the system is being poisoned by SEO-driven content or where it is hallucinating to satisfy a user bias. This requires the patience to look under the hood, a task most users avoid, creating a significant moat for those willing to do the hard work of technical auditing.

Key Action Items

• Audit Your Information Inputs: Over the next quarter, stop relying on chatbot summaries for critical decisions. Re-adopt the practice of visiting primary sources directly to bypass the flattened consensus generated by AI.
• Adopt the Design Stance: When an AI provides an answer, stop asking what it means and start asking how it was optimized. Identify the likely training data or marketing incentives that shaped the response.
• Implement Human-Speed Verification: For important projects, force a 24-hour cooling-off period between receiving AI-generated insights and acting on them. This breaks the Gish gallop effect and allows for deliberate, rather than reactive, thinking.
• Diversify Your Information Diet: Actively seek out non-AI-indexed sources, such as books, niche forums, and direct expert interviews, to maintain a baseline of reality that is not susceptible to GEO manipulation. This is a long-term investment in cognitive independence.
• Monitor for Bot-Poisoning: If you are a content creator, ensure your work is not just optimized for crawlers. Focus on depth and unique synthesis that bots cannot easily replicate or boil down into a two-sentence answer. This pays off in 12-18 months by building a brand that remains distinct from the AI-generated noise.