AI Drives Storage Apocalypse -- Blu-ray Offers Archival Solution
TL;DR
- The "storage apocalypse" is driven by AI data centers consuming vast amounts of storage, leading to backorders of up to 24 months and projected price increases through 2027.
- Optical media, specifically Blu-ray discs, offers a cost-effective alternative at approximately 3 cents per gigabyte, providing a viable solution for long-term archival storage unaffected by AI-driven market demand.
- The increasing cost and decreasing availability of consumer-grade SSDs and HDDs are forcing users to re-evaluate their digital hoarding habits and consider more aggressive data pruning strategies.
- Blu-ray discs, with their potential 26-year lifespan and low cost per gigabyte, present a practical, albeit labor-intensive, solution for preserving data that does not require immediate access.
- The BlueVault application simplifies Blu-ray archiving by managing disc labeling, file integrity checks, and multi-disc sets, enabling users to create accessible archives without proprietary software dependencies.
- The shift in storage economics necessitates a re-evaluation of backup strategies, potentially requiring increased redundancy, more robust RAID configurations, and a greater reliance on used or specialized storage markets.
Deep Dive
The current surge in storage prices, driven primarily by AI infrastructure demand, is creating a "storage apocalypse" that will persist through at least 2027. This price inflation, affecting both SSDs and HDDs, is forcing individuals and organizations to re-evaluate their data storage strategies, moving beyond simple capacity expansion to more creative, cost-effective, and potentially older methods.
The core implication of this storage crunch is a fundamental shift in how users manage digital assets. For many, the automatic impulse to "keep everything" is no longer economically viable. This necessitates a more deliberate approach to data curation, leading to the deletion of redundant or infrequently accessed files, as exemplified by the discussion around deleting old video project files that are unlikely to be reused. Furthermore, the rising cost of new storage is making previously overlooked or deprecated solutions increasingly attractive. Optical media, particularly Blu-ray discs, is re-emerging as a viable option due to its significantly lower cost per gigabyte compared to SSDs and HDDs, offering a potential long-term storage solution for media and archival data that data centers are not prioritizing. This necessitates a return to older, more manual methods of data management, including the use of physical media and the development of tools to manage these archives effectively.
The long-term consequence of this price shock is a potential re-evaluation of data retention policies and a renewed appreciation for efficient data management practices. While the immediate impact is financial pressure, the downstream effect may be a more mindful approach to digital hoarding and a greater adoption of solutions that prioritize cost-effectiveness and longevity over raw speed or convenience. The development of tools like "BlueVault" reflects this emergent need for specialized software to manage archival data on optical media, highlighting a gap in current backup solutions that cater to this specific, cost-driven requirement.
Action Items
- Audit storage acquisition: Analyze 3-5 potential bulk purchase options for SSDs and HDDs to mitigate price volatility.
- Implement optical media archiving: Create a system for archiving 50GB-100GB Blu-ray discs for long-term data preservation.
- Develop data retention policy: Define criteria for deleting 20-30% of non-essential digital assets to reduce storage needs.
- Evaluate backup strategy: Compare cloud backup providers (e.g., Backblaze) against local solutions for 3-5 critical data sets.
- Research used enterprise storage: Investigate 2-3 vendors specializing in refurbished SAS drives for cost-effective capacity.
Key Quotes
"The storage apocalypse has arrived. It's here, it's real, and an old friend's going to drop by to talk survival strategies as prices explode."
This quote introduces the central theme of the podcast episode: the significant and ongoing increase in storage prices. The hosts frame this as an "apocalypse," indicating the severity of the situation and setting the stage for discussions on how to cope with these rising costs.
"I predict that the linux desktop market share will become 5% at some point this year. I think that one was right too, right? Like, everything we can measure with seems like it hit, which is great."
Michael Tunnell reflects on his past predictions, highlighting the accuracy of his forecast regarding Linux desktop market share reaching 5%. This demonstrates a successful prediction and suggests a positive trend for Linux adoption on the desktop.
"The worst price increases seem to be in like, the kind of nicer SSDs and NVMe's in the two terabyte and up range. I don't see a lot of punishment in the one terabyte, but for example, the Western Digital Black SN850X 2 terabyte drive is up 80% since October."
Brent details the specific areas of the storage market experiencing the most dramatic price hikes. He points to higher-capacity SSDs and NVMe drives as being particularly affected, citing a significant percentage increase for a popular model.
"So the seed oil of this idea was after last week's Linux Unplugged, West and I were checking out the DVD drive that I bought to restore or re-rip stuff from the original DVD since I deleted and then wanted it back."
Chris explains the origin of his idea to revisit optical media for storage solutions. This anecdote about needing to re-access deleted content and the subsequent search for a DVD drive highlights a personal motivation for exploring older storage technologies.
"BlueVault is available at github.com/ChrisLAS/blue-vault. It is vibe-coded. Warning: I did notice I don't think you can type 'r' in. Yes, I fixed that."
Chris introduces his new open-source application, BlueVault, designed for managing Blu-ray archives. He provides its GitHub repository and notes a specific bug fix, indicating ongoing development and community engagement.
"Kopia is a cross-platform backup tool for the various operating systems out there with some really nice features. I've been diving into Kopia; it was mentioned a couple of times by some of our listeners as a response to our, you know, several scattered backup episodes that we've done in the past."
Brent highlights Kopia as a recommended backup tool, noting its cross-platform compatibility and positive reception from the audience. He explains that it was suggested in response to previous discussions about backup strategies.
Resources
External Resources
Books
- "The Cosmic Desktop" by System76 - Mentioned as a prediction for a release candidate version in 2025.
Articles & Papers
- "Death of Stack Overflow" - Discussed as an example of information that might vanish from the internet.
People
- Michael Tonell - Guest, from Tux Digital and This Week in Linux.
- Chris - Host.
- Wes - Host.
- Brent - Host.
- Jason - Mentioned as a professional D&D host.
Organizations & Institutions
- System76 - Developer of the Cosmic Desktop.
- KDE - Mentioned in relation to a potential desktop operating system release and the Telco project.
- Define Networking - Sponsor of the unplugged program.
- Flocks - Facilitating the Planet Nix event.
- Planet Nix - An event focused on reproducible dev environments.
- Scale - An event that Planet Nix leads into.
- Linux Fest Northwest - An upcoming event.
- New England Patriots - Mentioned as an example team for performance analysis.
- Pro Football Focus (PFF) - Data source for player grading.
- Universal Blue Project - Mentioned for its effort to make systems destroless.
- Western Digital - Manufacturer of the Black SN850X SSD.
- Newegg - Retailer mentioned for Blu-ray disc pricing.
- Crowdhealth - A health insurance alternative.
- Fountain FM - A platform for boosting podcasts.
- Podcast Apps - A platform offering additional podcast features.
- Unifi - Mentioned in relation to monitoring capabilities.
- Foundry VTT - Used for virtual tabletop gaming.
- Jitsi - Used for player communication in tabletop gaming.
- Jupiter Broadcasting - Host of the unplugged program.
Websites & Online Resources
- Define Net Unplugged - A decentralized VPN built on the open-source Nebula project.
- GitHub - Platform where Nebula project development is visible.
- CamelCamelCamel - Used for comparison shopping of SSD vendors.
- GitHub.com/chrislass/bluevault - Repository for the Blue Vault application.
- cdemu.org - Website for the CD-EMU emulation tool.
- Linux Unplugged.com - Website for the Linux Unplugged program.
- Jupiterbroadcasting.com - Website for Jupiter Broadcasting.
Other Resources
- Nebula Project - Open-source project underlying Define Net Unplugged.
- Cosmic Desktop - Desktop environment from System76.
- Linux Desktop Market Share - Discussed in relation to predictions.
- KDE Linux - A potential desktop operating system from KDE.
- Storage Apocalypse - A term used to describe the current state of storage prices.
- SSDs (Solid State Drives) - Mentioned in relation to price increases.
- NVMe Drives - Mentioned in relation to price increases.
- NAS (Network Attached Storage) - Discussed in relation to storage needs.
- AI Jobs - Cited as a reason for increased storage demand.
- Magnetic Tape - Mentioned as a potential storage solution.
- Optical Media - Discussed as a solution for archival storage.
- Blu-ray Discs - Mentioned as a cost-effective archival storage medium.
- M-Disc - A type of Blu-ray disc with claimed longevity.
- Borg Backup - A backup tool.
- Bacula - A backup tool.
- Blue Vault - A Rust-based GUI application for managing Blu-ray archives.
- Power ISO - An application for burning discs and creating file systems.
- Telco Project - A KDE project for tracking collections.
- CD-EMU - A tool for emulating optical drives.
- Cloud Chapters - A feature for navigating podcast episodes.
- Transcripts - Available for podcast episodes.
- Net Alert X - A tool for monitoring network devices.
- D&D (Dungeons & Dragons) - A tabletop role-playing game.
- Nextcloud - Used for content hosting in tabletop gaming.