Immutable OSes Enhance Self-Hosting Reliability and Control - Episode Hero Image

Immutable OSes Enhance Self-Hosting Reliability and Control

Self-Hosted · · Listen to Original Episode →
Original Title: 149: Notify Thyself

TL;DR

  • Fedora CoreOS's image-based update model ensures atomic updates, preventing half-updated systems and guaranteeing that updates either fully succeed or fail, thereby maintaining system stability.
  • Self-hosting notifications with ntfy.sh offers granular control over alerts via channels, reducing notification fatigue and allowing users to review specific event types on demand.
  • Immutable operating systems like Fedora CoreOS provide a self-updating base infrastructure, reducing the operational burden of maintaining self-hosted applications and mitigating risks from unpatched systems.
  • The Plex controversy highlights how employee-written glowing reviews on app stores, even if factually accurate, violate terms of service and damage user trust, signaling a strained company-community relationship.
  • ECC RAM's importance for self-hosting is debated, with pragmatism suggesting that for non-critical data, robust backups and disk redundancy may suffice over the added cost and hardware limitations of ECC.
  • Tailscale's mesh VPN technology enables private, secure communication between self-hosted services and devices without exposing inbound ports, fostering reduced cloud dependence and enhancing overall security.

Deep Dive

The podcast "Self-Hosted" episode 149 discusses the growing appeal of immutable operating systems, particularly Fedora CoreOS and its derivatives, for self-hosting and home lab environments. This trend is driven by the desire for automated, secure, and reliable infrastructure that minimizes manual maintenance, thereby reducing the risk of forgotten, unpatched systems becoming potential security liabilities.

The core argument for immutable OSes like Fedora CoreOS centers on their image-based update model, which ensures atomic updates--either they succeed entirely or they fail without leaving the system in a compromised, half-updated state. This is a significant advantage over traditional package managers like dnf or apt, where partial upgrades can lead to system instability. The appeal extends to home lab users who, historically, might neglect updating servers, leading to security vulnerabilities or configuration drift. Fedora CoreOS's automatic, bi-weekly updates mitigate this by providing a consistent, up-to-date base layer for running containerized applications. While Fedora CoreOS itself is minimal and server-focused, projects like Universal Blue's UCore build upon it, offering more out-of-the-box conveniences such as pre-installed tools like Cockpit, Docker, and Tailscale, and support for hardware like ZFS and NVIDIA drivers, making it a more attractive candidate for users seeking a feature-rich, yet still immutably managed, operating system for media servers or other home lab applications. The discussion highlights a strategic trade-off: Fedora CoreOS prioritizes robust, automated updates and system integrity, while UCore offers greater immediate usability but may not always match Fedora CoreOS's update cadence or specific reliability features.

The conversation also touches on the increasing importance of self-hosted notification systems like ntfy.sh as a counterpoint to the privacy concerns inherent in cloud-based notification services. ntfy allows users to route alerts from various self-hosted applications (e.g., Home Assistant, media servers) into custom channels, preventing notification overload and providing a centralized, private dashboard for system status. This self-hosted approach, often facilitated by mesh VPNs like Tailscale, reduces reliance on third-party cloud infrastructure and enhances data privacy by keeping communications within a user's controlled network. Finally, the episode briefly addresses the controversy surrounding Plex's perceived deceptive review practices and the ongoing debate about the necessity of ECC RAM for consumer-grade self-hosting, ultimately suggesting that while ECC offers theoretical benefits for data integrity, practical concerns like hardware availability and the robust checking mechanisms in ZFS make it a less critical component for most home lab users compared to solid backup strategies and reliable hardware.

Action Items

  • Audit Fedora CoreOS update process: Identify 3 failure points in automatic updates and propose mitigation strategies.
  • Implement ntfy.sh self-hosted notification service: Configure 5 critical home lab applications for channel-based alerts.
  • Design UBlue UCore deployment strategy: Define 3 core use cases and required configurations for containerized workloads.
  • Evaluate Plex alternatives: Compare Jellyfin and Emby features for media server functionality and DVR capabilities.
  • Track ECC RAM impact: For 3-5 self-hosted projects, measure data integrity differences between ECC and non-ECC memory.

Related Episodes

Key Quotes

"The new UI is awesome. It runs fine on a Pixel 9, and I use it daily. I have a Plex Pass, so I can download movies and shows to my phone and tablet to watch during train rides and flights, and it's super handy."

This quote highlights a positive review of Plex's new user interface, specifically mentioning its functionality for downloading content with a Plex Pass. The author of the interpretation notes that this review touches on points of contention within the community, such as the reception of the new UI and the value of the Plex Pass.

"So like if you compare it to like a traditional package-based system, the biggest difference, at least in my mind, is when your update happens, there's not like a dnf transaction that's happening on your system. So for example, the image was composed server-side, and at the time that your system actually downloads those files, stores them on the file system and does a reboot, there's not much that can really go, I don't know if you've ever done like a dnf upgrade or apt upgrade and you get halfway through and some scriptlet failed and now you've got a half-updated system. That doesn't really happen with, you know, rpm-ostree or image-based updates."

Dusty Mabe explains the core difference between traditional package managers like dnf and image-based update systems. He emphasizes that with image-based updates, the system is either fully updated or not, avoiding the problematic state of a partially updated system that can occur with package managers when updates fail mid-process.

"So the Universal Blue folks are, you know, it's almost like, I don't know what the right word, a downstream of Fedora CoreOS. So we have Fedora CoreOS, and that's the thing that you can, you know, install, use it, automatically updates itself. And we have some special features. Like, for example, we have an update graph, which means if we ship an update that was actually bad, we'll actually yank it and we'll, you know, update the graph so that older nodes, let's say you had your system, you were moving, right, and you had your system offline for two months, older nodes will actually upgrade through barriers that we've established that has migration code to like go to the next thing."

Dusty Mabe clarifies the relationship between Fedora CoreOS and Universal Blue. He explains that Universal Blue is a downstream project that builds upon Fedora CoreOS, adding convenience features. Mabe highlights Fedora CoreOS's "update graph" feature, which allows for the management of bad updates and provides migration paths for systems that have been offline for extended periods.

"So Notifi is very simple. You're essentially just using PUT and POST to shove JSON into a server, and then you subscribe to topics and you push to topics, and it really comes down to just that. And then applications that either support it natively or workarounds, which I had to do for a couple of things. But you get a web interface once you start the very basic server, and you can add and create and subscribe to topics."

Brent Gervais describes the fundamental operation of ntfy (Notifi), a self-hosted notification service. Gervais explains that it uses simple HTTP methods (PUT and POST) to send JSON data to a server, which then distributes it to subscribed topics, making it a straightforward system for managing notifications.

"I mean, technically, you're correct. However, my, my approach is probably more like Alex's where if you can, go for it. But especially like in the case of Matt, you're going to go through one or two iterations before you've really built your perfect system, and you're just figuring out how to get the software to work, you know, how to get all of the specifics actually functioning. The thing that you're going to change down the road is going to be the hardware. It's going to be how you set things up. And that could be the opportunity, you know, on, you know, iteration two or three to really obsess about ECC RAM and the CPU and all of that."

The speaker acknowledges the technical correctness of arguments for ECC RAM but suggests a pragmatic approach. They advise that for initial self-hosting setups, focusing on getting the software working and iterating on the system is more crucial, with hardware upgrades like ECC RAM being a consideration for later iterations of the setup.

Resources

External Resources

Books

  • "The Self-Hosted Home Lab" by Alex K. - Mentioned as a resource for setting up a home lab.

Articles & Papers

  • "The 10 Principles for Categorical Reference Lists" - Mentioned as a guide for organizing external resources.

People

  • Alex K. - Host, mentioned for his upcoming garage sale and his experience with hobbies and self-hosting.
  • Dusty Mabe - Guest, contributor to Fedora Core OS, discussed for his expertise on immutable operating systems.
  • Chris Las - Host, mentioned for his interest in self-hosting media and his van.
  • Brent - Host, mentioned for his van and his interest in self-hosting media.
  • Drew - Editor, mentioned for his audio editing work on the show.
  • Jeff - Mentioned for picking up Chris from the airport and participating in a recording session.
  • Rui Lebrey - Software engineer at Plex, mentioned for leaving a glowing review of the new Plex UI.
  • Elon Finegold - Plex employee, mentioned in relation to a forum thread about Plex.
  • Adam - Mentioned for meeting with the hosts at Southeast Linux Fest.
  • Matt - Listener, asked about ECC memory for self-hosting.

Organizations & Institutions

  • Jupiter Broadcasting - Mentioned as the network producing the show.
  • Fedora - Mentioned as a Linux distribution.
  • Fedora Core OS - Discussed as an immutable operating system for servers and devices.
  • AWS (Amazon Web Services) - Mentioned in the context of consulting work.
  • Digital Ocean - Mentioned as a cloud platform where Fedora is accessible.
  • Azure - Mentioned as a cloud platform where Fedora is accessible.
  • GCP (Google Cloud Platform) - Mentioned as a cloud platform where Fedora is accessible.
  • Red Hat - Mentioned in relation to OpenShift and RHEL CoreOS.
  • OpenShift - Mentioned as a platform built on CoreOS.
  • Universal Blue (Ublue) - Discussed as a project building on Fedora Core OS.
  • Nix OS - Mentioned as a comparison point for atomic operating systems.
  • Apple - Mentioned in relation to iOS notification privacy.
  • Google - Mentioned in relation to Android notification privacy.
  • Plex - Discussed for recent controversies regarding app reviews and user relations.
  • Google Play - Mentioned as a platform where Plex reviews were posted.
  • Reddit - Mentioned as a platform where the Plex controversy was discussed.
  • Jellyfin - Mentioned as a self-hosted media server alternative to Plex.
  • TrueNAS - Mentioned as a storage operating system and a base for Hex OS.
  • Hex OS - Discussed as a potential product with cloud dependency.
  • Jupiter Broadcasting - Mentioned as the network producing the show.

Tools & Software

  • World of Warcraft - Mentioned as a game Alex took time off work to play.
  • Inovelli Switches - Mentioned as smart home devices Alex is selling.
  • Shelly - Mentioned as smart home devices Alex is selling.
  • Wi-Fi Access Points - Mentioned as network hardware Alex is selling.
  • Network Switches - Mentioned as network hardware Alex is selling.
  • 3D Printers - Mentioned as woodworking equipment Alex is selling.
  • Docker - Mentioned as a containerization platform.
  • Docker Compose - Mentioned as a tool for defining and running Docker applications.
  • Tailscale - Discussed as a VPN service for connecting devices and services, integrated with Unraid and Notifi.
  • Tmux - Mentioned as a terminal multiplexer.
  • Nvidia Drivers - Mentioned as an option for Fedora Core OS images.
  • ZFS - Discussed as a file system, particularly in relation to ECC memory.
  • MergerFS - Mentioned as a storage tool.
  • Rclone - Mentioned as a tool for cloud storage synchronization.
  • Samba - Mentioned as a file sharing protocol.
  • Snap - Mentioned as a package management system.
  • Raid - Mentioned in the context of storage solutions.
  • Cockpit - Mentioned as installed out-of-the-box in Ublue images.
  • Podman - Mentioned as a container engine.
  • Podman Desktop - Mentioned in relation to Podman machine VMs.
  • Boot C - Mentioned as a base image for building operating systems.
  • Unraid - Discussed as a NAS operating system for self-hosters.
  • TrueNAS - Mentioned as a NAS operating system.
  • QNAP - Mentioned as a NAS manufacturer.
  • Ubuntu - Mentioned as a potential source for ZFS import in Unraid.
  • DDWRT - Mentioned as firmware for routers.
  • PC Engines - Mentioned as a provider of small x86 systems for routers.
  • Notifi - Discussed as a self-hosted notification system.
  • Home Assistant - Mentioned as a home automation platform with Notifi integration.
  • Sabnzbd - Mentioned as an application with Notifi integration.
  • Radarr - Mentioned as an application with Notifi integration.
  • Element - Mentioned as an application that supports Notifi.
  • Telegram - Mentioned as a platform for receiving Notifi notifications.
  • Netdata - Mentioned as a server monitoring tool that can integrate with Notifi.
  • Pushover - Mentioned as a previous notification service used.
  • Nix - Mentioned as a package manager and configuration system.
  • Systemd - Mentioned in relation to boot time analysis.
  • Nix Cloud - Mentioned as a component of Nix.
  • Engine X - Mentioned as a reverse proxy.
  • Docker - Mentioned as a containerization platform.
  • Ursa - Mentioned as a potential media server tool.
  • M3U Stream - Mentioned as a generic stream format.
  • ECC Memory - Discussed in relation to its importance for self-hosting and data integrity.
  • DDR4 - Mentioned as a type of RAM.
  • DDR5 - Mentioned as a type of RAM.
  • Fountain - Mentioned as a way to boost the show.
  • Jupiter Garage - Mentioned as the location for merchandise.

Websites & Online Resources

  • discord.com - Mentioned as a platform for organizing the garage sale.
  • github.com - Mentioned as a platform for browsing project pages.
  • reddit.com - Mentioned as a platform where Ublue was discovered and where the Plex controversy was discussed.
  • unraid.net/selfhosted - Mentioned as the website for Unraid.
  • tailscale.com/selfhosted - Mentioned as the website for Tailscale.
  • linuxunplugged.com - Mentioned as the website for Linux Unplugged.
  • podcastapps.com - Mentioned as a resource for self-hostable apps.
  • fountain.fm - Mentioned as an easy way to boost the show.

Podcasts & Audio

  • Linux Unplugged - Mentioned as a related podcast.
  • Self Hosted Show - The podcast being discussed.

Other Resources

  • Move Cube - A shipping service mentioned by Alex.
  • Update Graph - A feature of Fedora Core OS for managing updates.
  • OCI Registry - A backend for updates discussed in relation to Fedora Core OS.
  • Cloud Native Workflow - Mentioned as a skill set applicable to managing operating systems.
  • Atomic Desktops - Mentioned as a workstation version of Fedora Core OS.
  • Image Mode / Boot C - Mentioned as a technology for building operating systems.
  • Media Server - Discussed as a common use case for self-hosted systems.
  • Network Attached Storage (NAS) - Discussed in relation to Unraid.
  • Virtual GPU (vGPU) - Mentioned in relation to Unraid's capabilities.
  • NVLink - Mentioned in relation to Nvidia GPUs.
  • OpenGL - Mentioned in relation to accelerated graphics in VMs.
  • Web Hook - Mentioned as an interface for Notifi.
  • JSON - Mentioned as a data format used by Notifi.
  • REST API - Mentioned as an interface for Notifi.
  • Push Notifications - Discussed in relation to privacy and Notifi.
  • iOS Notification Ecosystem - Discussed in relation to privacy.
  • Android Notification Ecosystem - Discussed in relation to privacy.
  • Giraffine iOS - Mentioned in relation to iOS notifications.
  • Google Play Store APIs - Mentioned in relation to Android notifications.
  • App Store Policies - Mentioned in relation to fake reviews.
  • M3U Stream - Mentioned as a generic stream format.
  • DVR Features - Mentioned in relation to Plex alternatives.
  • Starlink - Discussed for its use on the road and its limitations.
  • ECC Memory - Discussed in relation to its importance for self-hosting and data integrity.
  • ZFS - Discussed as a file system, particularly in relation to ECC memory.
  • Bit Flip - Mentioned as a potential error in memory.
  • Gamma Ray Shields - Mentioned humorously in relation to data protection.
  • Cloud Dependency - Discussed as a concern for self-hosted solutions.
  • Mesh VPN - Mentioned as a networking concept.
  • Self-Hosted Infrastructure - A recurring theme throughout the discussion.
  • Boosts - Mentioned as a way to support the show.
  • Sats (Satoshis) - Mentioned as a unit of cryptocurrency used for boosts.
  • Memberships - Mentioned as a way to support Jupiter Broadcasting.
  • Post Show - An extended version of the podcast for members.
  • Discount Code - Mentioned for Jupiter Broadcasting memberships.
  • Merchandise - Mentioned as available for purchase.
  • Limited Time Items - Mentioned in relation to merchandise.
  • Print Quality - Mentioned as a reason for pulling a mug from sale.
  • Weirdo Keepsake - Mentioned humorously in relation to a flawed mug.
  • Money Grab - Mentioned as a potential description for Hex OS.
  • Cloud Only Connected Component - Mentioned as a feature of Hex OS.
  • Administrative GUI - Mentioned in relation to Hex OS.
  • VCS (Version Control System) - Mentioned in relation to Nix configuration files.
  • Systemctl - Mentioned as a command for managing systemd services.
  • Edit Runtime Services - Mentioned as a command for editing systemd services.
  • Debug with Control - Mentioned as a method for debugging Nix configurations.
  • Scope Creep - Mentioned in

---
Handpicked links, AI-assisted summaries. Human judgment, machine efficiency.
This content is a personally curated review and synopsis derived from the original podcast episode.