Google's Transformer Dilemma: Inventing AI, Losing the Race

Resources

Books

• "In The Plex" by Steven Levy - A source for information about Google's early history and internal workings.
• "Supremacy" by Parme Olson - Explores the history of DeepMind and OpenAI.
• "Genius Makers" by Cade Metz - Covers the history of AI and its key players.

Videos & Documentaries

• Jeff Hinton's Tech Talk at Google - An artifact of history available on YouTube, detailing his work on neural networks.
• AlphaGo Documentary - Produced by Google, it tells the story of DeepMind training a model to beat the world Go champion.

Research & Studies

• The Transformer Paper (Vaswani et al., 2017) (Google Brain team) - This foundational paper introduced the Transformer architecture, which revolutionized NLP.
• "Building High-Level Features Using Large-Scale Unsupervised Learning" (The Cat Paper) (Google Brain team, 2011) - Demonstrated that large neural networks could learn meaningful patterns without supervision, running on a distributed system.
• "The Bitter Lesson" by Rich Sutton (2019) - Argues that scalable architectures combined with more data and compute are the primary drivers of AI progress.

Tools & Software

• TensorFlow - An open-source machine learning framework developed by Google Brain to enable researchers to build and train ML models.
• Distbelief - A distributed system developed by Jeff Dean for training large neural networks asynchronously.
• Kubernetes - An open-source container orchestration system developed by Google to make applications more portable across clouds.
• TPU (Tensor Processing Unit) - Google's custom ASIC designed for neural network matrix multiplication.

Articles & Papers

• "Attention Is All You Need" (Google Brain team, 2017) - The seminal paper that introduced the Transformer architecture.

People Mentioned

• Larry Page (Co-founder of Google) - Always viewed Google as an AI company and had a vision for its AI capabilities.
• Sergey Brin (Co-founder of Google) - Collaborated with Larry Page on Google's AI vision and strategy.
• George Herrick (Google Engineer) - Proposed the theory that compressing data is equivalent to understanding it.
• Ben Gomes (Google Engineer) - Discussed the data compression theory with George Herrick.
• Noam Shazeer (Google Engineer) - Co-authored the Transformer paper and played a key role in its development and implementation.
• Sanjeev Gemawat (Google Engineer) - Supported George Herrick and Noam Shazeer's work on language models.
• Susan Wojcicki (Former CEO of YouTube) - Borrowed the language model "Phil" for AdSense development.
• Jeff Dean (Google Senior Fellow) - A central figure in Google's AI infrastructure and development, instrumental in optimizing systems and developing key technologies.
• Chuck Norris - Mentioned in the context of "Jeff Dean facts."
• Urs Hölzle (Google Senior VP, Technical Infrastructure) - Worked with Jeff Dean on Google's infrastructure.
• Sebastian Thrun (Former Head of Google X, Stanford Professor) - Recruited to Google to lead AI applications, including Project Ground Truth and Waymo.
• Andrew Ng (Computer Science Professor, Former Head of Google Brain) - Recruited to Google to lead the Google Brain project.
• Greg Corrado (Neuroscience PhD, Researcher at Google) - Co-founder of Google Brain, worked on the "Cat Paper."
• Sundar Pichai (CEO of Google and Alphabet) - Made key strategic decisions regarding Google's AI direction and product launches.
• Elon Musk (Founder of SpaceX, Tesla, etc.) - Invested in DeepMind and co-founded OpenAI, concerned about AI safety.
• Sam Altman (CEO of OpenAI, Former President of Y Combinator) - Co-founded OpenAI and led its transition to a for-profit structure.
• Reid Hoffman (Co-founder of LinkedIn, Investor) - Board member at Microsoft and OpenAI, played a role in Microsoft's investment in OpenAI.
• Demis Hassabis (CEO of DeepMind) - Co-founder of DeepMind, a key figure in AI research and development.
• Shane Legg (Co-founder of DeepMind) - Co-founder of DeepMind, popularized the term Artificial General Intelligence (AGI).
• Mustafa Suleyman (Co-founder of DeepMind, Head of Inflection AI) - Co-founder of DeepMind, later shifted to Google AI policy and then co-founded Inflection AI.
• Ray Kurzweil (Futurist, Former Google Chief Futurist) - Organized the Singularity Summit where DeepMind sought funding.
• Peter Thiel (Co-founder of PayPal, Investor) - Led DeepMind's seed round funding through Founders Fund.
• Mark Zuckerberg (CEO of Meta) - Attempted to acquire DeepMind for Facebook.
• Yann LeCun (AI Researcher, Former Postdoc of Jeff Hinton) - Co-founder of FAIR (Facebook AI Research).
• Alex Krizhevsky (AI Researcher, Co-founder of DNN Research) - Co-creator of AlexNet.
• Ilya Sutskever (AI Researcher, Co-founder of DNN Research, Former Chief Scientist of OpenAI) - Co-creator of AlexNet and a key figure in OpenAI's founding.
• Jensen Huang (CEO of NVIDIA) - Mentioned in relation to the impact of AlexNet on NVIDIA's trajectory.
• George Boole - Jeff Hinton's great-great-grandfather, inventor of Boolean algebra.
• Satya Nadella (CEO of Microsoft) - Led Microsoft's strategic partnership and investment in OpenAI.
• Toby Lutke (Founder and CEO of Shopify) - Discussed on the Acquired podcast episode ACQ2.
• J.P. Morgan Payments - Presented partner for the podcast.
• Century - A partner for the podcast, helping developers debug software.
• Work OS - A partner for the podcast, providing enterprise readiness solutions for software.
• Shopify - A partner for the podcast, a leading commerce platform.
• Hamilton Helmer - Author of "7 Powers," a framework for analyzing business advantages.
• Bill Gross - Pointed out the difference in query length between traditional search and AI chat.
• Brett Taylor - Recommended a documentary about Waymo.
• Ravi Gupta - Partner at Sequoia, co-host of The Glue Guys podcast.
• Shane Battier - Former basketball player, co-host of The Glue Guys podcast.
• Alex Smith - Former NFL quarterback, co-host of The Glue Guys podcast.
• Wright Thompson - Author interviewed on The Glue Guys podcast.
• Bill Gurley - VC known for his writing and accent, compared to Wright Thompson's.
• Nando Shazair - Mentioned as a former Google employee and AI talent.
• Demis Hassabis - CEO of DeepMind, childhood chess prodigy.
• Jeff Hinton - Godfather of neural networks.
• Cade Metz - Author of "Genius Makers."
• Jim Gao - Friend of the show, led a data center cooling project at Google.
• Chaitanya Puttagunta - Partner at Benchmark.
• Dwarkash Patel - Helped the host with conclusions.
• Brian Lawrence - From Oak Cliff Capital, helped with AI data center economics.
• Ben Thompson - Author of "Stratechery," discussed the "YouTube internet" and AI.
• Gavin Baker - Partner at Andreessen Horowitz, studied AI chip economics.
• M.G. Siegler - Writer at TechCrunch.
• Ben Idelson - Thought partner for the episode, host of the Step Change podcast.
• Koray Kavukcuoğlu - From the DeepMind team, worked on Gemini.
• Shishir Mehrotra - CEO of Grammarly, formerly at YouTube.

Organizations & Institutions

• Google Brain - Google's AI research team responsible for significant breakthroughs.
• DeepMind - Acquired by Google, a leading AI research lab.
• OpenAI - A prominent AI research lab, known for GPT models.
• Anthropic - An AI safety and research company.
• NVIDIA - A leading designer of GPUs essential for AI development.
• Microsoft - Strategic partner and investor in OpenAI.
• Stanford University - Affiliation of Sebastian Thrun and other key AI researchers.
• University of Toronto - Affiliation of Jeff Hinton and his students.
• Darpa - Defense Advanced Research Projects Agency, sponsored challenges spurring AI research.
• Founders Fund - Venture capital firm that invested in DeepMind.
• Alphabet - Google's parent company.
• Google X (X) - Google's moonshot factory.
• Y Combinator - Startup accelerator.
• Amazon Web Services (AWS) - Cloud computing platform.
• NIPS (now NeurIPS) - A leading machine learning conference.
• Imagenet Competition - A computer vision competition that spurred advancements in image recognition.
• Mechanical Turk - Amazon's crowdsourcing marketplace, used for labeling Imagenet data.
• NVIDIA GTC - NVIDIA's annual developer conference.
• Broadcom - Chip manufacturer.
• TSMC - Semiconductor foundry.
• Oracle - Enterprise software company.
• Silver Lake - Private equity firm.
• Canada Pension Plan Investment Board - Institutional investor.
• Andreessen Horowitz - Venture capital firm.
• Mubadala - Abu Dhabi's sovereign wealth fund.
• NFL - National Football League, hosting an innovation summit.
• Sequoia Capital - Venture capital firm.
• Step Change Podcast - A podcast recommended by the hosts.
• The Glue Guys Podcast - A podcast recommended by the hosts.

Courses & Educational Resources

• N/A

Websites & Online Resources

• Acquired.fm - The podcast's website, offering email lists, Slack community, and show notes.
• Acq2 - The podcast's interview show.
• YouTube - Platform where Jeff Hinton's tech talk is available.
• Amazon Prime Video - Where the Waymo documentary is available.
• Wired - Publication that featured articles on Google's AI history.

Other Resources

• Atari Game Breakout - A game used by DeepMind to demonstrate AI capabilities.
• Dota 2 - A video game used by OpenAI for AI research.
• StarCraft - A real-time strategy game used by DeepMind for AI research.
• Atari Games - A collection of games used by OpenAI in their Universe project.
• Grand Theft Auto - An open-world game used by OpenAI in their Universe project.
• Rubik's Cube - A puzzle used by OpenAI to teach AI.
• F1 (Film) - A movie recommended by one of the hosts.
• Sea of Stars - An indie RPG recommended by one of the hosts.
• Travelpro Suitcase - A brand of luggage recommended by one of the hosts.