Data Engineering Podcast
By Tobias Macey
This show goes behind the scenes for the tools, techniques, and difficulties associated with the discipline of data engineering. Databases, workflows, automation, and data manipulation are just some of the topics that you will find here.
20 episodes
All Episodes
AI-First Loop Drives 10-50x Productivity in Autonomous Data Engineering
AI agents now execute complex data engineering tasks, delivering 10-50x productivity gains and redefining roles from coders to operators of intelligent systems.
View Episode Notes →
Treating Metering as a Financial System Drives Competitive Advantage
Treat metering as a financial system, not a data pipeline, to unlock competitive advantage. Accurate, granular usage data drives sophisticated pricing, proactive customer engagement, and prevents revenue leakage.
View Episode Notes →
Static PDFs Hinder Scientific Progress -- Composable Research Offers Solution
Static PDFs hinder scientific progress by locking discoveries in inaccessible formats. Embrace composable research for faster breakthroughs and a competitive edge.
View Episode Notes →
Building Adaptive AI Systems Through Feedback and Memory
True AI scalability hinges on adaptive systems and feedback loops, not just smart models. Build an evolving *system* for genuine competitive advantage.
View Episode Notes →
AI Analysts Drive Business Action Through Dynamic Contextual Dialogue
AI coworkers don't use magic; they bridge the gap between rapid human insight and slow data infrastructure by facilitating trustworthy dialogues that drive business action.
View Episode Notes →
Data Modeling Solves AI Hallucinations and Accelerates Delivery
AI amplifies data ambiguity, not fixes it. Prioritize enterprise data modeling to build trust and reliable insights, transforming reactive wrangling into proactive architecture.
View Episode Notes →
Crafting Dynamic AI Memory Beyond Simple Data Storage
AI memory is not about storing more data, but about storing it smarter. Build dynamic, context-aware AI systems by moving beyond simplistic RAG and vector stores to intelligent information management.
View Episode Notes →
Operationalizing LLMs: Observability, Prompt Management, and Cost Control
Unlock AI's true potential by mastering hidden operational complexities. Discover how robust systems, not just models, drive reliable LLM applications and avoid costly pitfalls.
View Episode Notes →
Modernization Enables AI Readiness Through Re-architecture
Legacy systems actively hinder AI innovation. Re-architect now to a document-first approach, unlocking agility and future-proofing your technology stack against accelerating AI demands.
View Episode Notes →
Database Version Control Enables Safe AI Agentic Writes
Databases become time machines and collaboration canvases with Git-style version control, reducing AI write risks and enabling reproducible ML for a competitive advantage.
View Episode Notes →
Business-Driven Semantic Models Prevent Data Architecture Entropy
Prioritize business-driven semantic models over physical data designs to prevent unmanageable systems and ensure trustworthy, reusable data assets.
View Episode Notes →
Lakehouse Observability: Petabyte-Scale Data Management and Interactive Troubleshooting
Unify petabyte-scale logs, metrics, and traces with lakehouse observability for low-latency, cost-efficient troubleshooting and new AI use cases.
View Episode Notes →
Fenic Integrates LLM Semantics Into DataFrame APIs For Efficient Data Engineering
Fenic embeds LLM semantics into data engineering, optimizing inference costs and enabling structured data extraction for more reliable, efficient AI pipelines.
View Episode Notes →
Data Teams Must Shift From Execution to Strategic Business Partnership
Data teams must shift from technical execution to strategic partnership by framing problems, developing perspectives, and driving actions, securing a seat at the decision-making table.
View Episode Notes →
F3: Decoupled Layout and WASM Enable Extensible Data Format
F3 decouples data layout and encodings, enabling self-decoding files with WebAssembly kernels for unparalleled extensibility and performance beyond Parquet's constraints.
View Episode Notes →
Metadata Platforms Evolve as Foundational Context Layers for AI Agents
Metadata platforms now empower AI agents with precise semantics, automating complex data management and governance for accurate, scalable AI outcomes.
View Episode Notes →
AI Transforms Data Engineering: New Assets, Testing, and Uptime Demands
AI transforms data engineering by processing unstructured data into new asset types like vectors and demanding continuous availability for interactive applications.
View Episode Notes →
Malloy: Human-Centric Data Interaction Beyond SQL
Malloy transforms data interaction, enabling intuitive exploration by treating SQL as an assembly language and preserving context for iterative analysis.
View Episode Notes →
AI Agents Accelerate Development, Shifting Bottlenecks to Orchestration
AI accelerates software development 2-10x, shifting bottlenecks to review and coordination. Master "context as code" and multiplayer agent environments to gain a significant velocity advantage.
View Episode Notes →
Temporal Platform Simplifies Resilient Application and Data Pipeline Development
Durable execution shifts retry, checkpointing, and error handling to the platform, freeing developers to focus on core business logic and dramatically boosting productivity for complex, stateful applications.
View Episode Notes →