The Hidden Infrastructure of Modern Research: Why Excel Hinders Discovery

In this conversation, gastroenterologist and researcher Shaun Chuah explains that the main bottleneck in medical research is not a lack of data, but a systemic reliance on fragile, manual tools like Microsoft Excel. By mapping the shift from manual sample tracking to an agentic AI operating system, Chuah highlights a core reality: the true competitive advantage in science today comes from building research-specific infrastructure that enforces data integrity and governance. For researchers and technical leads, the lesson is straightforward: stop treating data management as an afterthought. Those who invest in reproducible, automated pipelines now will outpace peers who remain stuck with the manual, error-prone workflows common in clinical research.

The Hidden Cost of Fast Solutions

Most research teams use Excel because it is immediate and requires no setup. However, Chuah’s experience shows that this fast start creates a compounding debt of operational complexity. When managing 30,000 samples across multiple cities, the lack of a structured database is not a minor inconvenience. It creates a systemic failure point where data becomes scattered across local drives, making reproducibility impossible.

"Most research teams do not really have a lab system because lab systems, enterprise software which is really expensive to procure it will take you like six months to set it up. And it is usually designed really for hospital operations where you are taking millions of samples and it is not really designed for a one off research study across a lot of different spaces."

-- Shaun Chuah

The transition to a custom Django application was not just a technical preference; it was a strategic choice to prioritize data integrity. By moving from manual spreadsheets to a system with built-in migrations and database management, the team created a foundation that could actually scale.

Where Immediate Pain Creates Lasting Moats

The most significant insight from the conversation is how Chuah leveraged agentic AI to bypass the manual labor of data analysis. Many teams view AI as a chatbot or a better search engine. Chuah’s approach, however, treats AI as an autonomous agent that interacts with tools like Python scripts and database queries to solve problems.

"The AI you give the AI a task, it looks at its tool set, it looks at what you are trying to do and then it goes away and works at it until it gives you an answer. And that is incredibly powerful."

-- Shaun Chuah

This creates a massive advantage. By building a sandbox where the AI operates on data within secure, compliant cloud environments, the team reduced tasks that previously took days of manual joining and filtering to mere minutes. This is not just efficiency; it is a fundamental change in the speed of discovery.

The System Responds: Governance as a Feature

A common failure in systems thinking is the belief that you can add security later. Chuah’s architecture integrates governance into the workflow. Because the AI is gated behind Django role-based access control, it cannot access sensitive data it should not see.

The system responds to the complexity of clinical research by ensuring that every AI action is verifiable. By forcing the AI to write and execute code to reach its conclusions, the researcher can audit the process. This creates a trust but verify loop that is essential for high-stakes medical research, where a wrong answer is a scientific failure.

Key Action Items

• Audit your Excel-dependency: Identify core research datasets currently trapped in spreadsheets. Over the next quarter, prioritize moving these into a structured database like SQL or Parquet to ensure data integrity.
• Implement CI/CD for research workflows: Stop manual deployments. Use GitHub Actions to automate your pipeline, allowing for rapid fixes and consistent environment management. This pays off in 6 to 12 months by preventing it works on my machine debugging cycles.
• Adopt agentic thinking: Stop using AI as a chatbot. Start designing tools, such as deterministic scripts, that your AI can call to perform specific, repeatable tasks. This investment creates a lasting advantage in your team throughput.
• Enforce data governance at the infrastructure level: If you are using AI, ensure the inference runs within a secure, compliant boundary, such as specific Azure or AWS data centers, rather than using public-facing APIs that risk data leakage.
• Focus on reproducibility over raw volume: As datasets grow into the terabytes, stop trying to store everything. Invest in defining the most valuable processed data and the pipelines required to recreate it. This is a 12 to 18 month investment that prevents long-term storage bloat.