Balancing Speed and Stability--Mitigating Tech Debt from Software Development Shortcuts

Key Quotes

"some of the best engineers out there are fundamentally lazy in that they will take the path of least resistance you know if you are measuring something they will try to gamefy that measurement and as long as the reports look good as long as it's easy and they're not getting bothered they are going to take whatever shortcut they want so yeah happy to get into specifics there but it's like water right that's going to flow wherever it's easiest to go"

Tom Totenberg argues that engineers often take the path of least resistance, which can lead them to exploit or "game" measurements if those measurements appear good and do not cause them immediate trouble. This tendency, Totenberg explains, is akin to water flowing downhill, seeking the easiest route.

"a concrete example of that is stuff like configuration management people have recognized the value of decoupling deployments which is the technical act of putting code into production or wherever you're deploying it right from releasing which is the business decision to expose new features and functionality to end users right those are two separate things deployment versus release and so i have seen so many weird cobbled together arcane strange configuration utilities out there that do operate in runtime that allow you to deploy darkly but then oh my gosh if anything goes off the rails it's an incident it's so tough to navigate and figure out the underlying root cause"

Totenberg highlights configuration management as a specific area where shortcuts can lead to problems, particularly when custom, "cobbled together" tools are used to decouple deployment from release. He explains that while these tools might allow for dark deployments (deploying code without making it visible to users), they can become difficult to troubleshoot during incidents, making root cause analysis challenging.

"something that i saw at a customer was they didn't modify their peer review process so a human so ryan you could go tell claude hey go generate some code work on this thing and then great it's going to get a peer review ryan you're a peer you can review that so you can tell it to create something the ai creates the code you review it no other human sees that and so being able to even circumvent that shortcut even in this forward thinking organization of saying okay if it's ai generated code we need at least two humans because that will make sure that one person isn't going to be the person who shoves something out the door with the assistance of claude or gemini or whatever llm you're relying on"

Totenberg describes a concerning shortcut observed with AI code generation where a peer review process was not adequately adapted. He explains that if a human reviews AI-generated code, and that human is the only reviewer, it bypasses the intended safety net, allowing a single person to potentially push unvetted code with AI assistance. Totenberg suggests that requiring at least two human reviewers for AI-generated code is necessary to mitigate this risk.

"so tech debt i think is a result of shortcuts right shortcuts can absolutely lead to tech debt and tech debt comes in multiple forms right whether this is something like wrapping code in some extra debugging observability wrapping code in feature flag or tech debt can also absolutely be the concept of someone trying to get the mvp out the door without thinking about the underlying fundamental platform that maybe should support that feature"

Totenberg clarifies that technical debt is a consequence of taking shortcuts. He explains that tech debt can manifest in various ways, such as adding extra debugging or feature flags, or more fundamentally, by prioritizing the Minimum Viable Product (MVP) without considering the underlying platform's long-term supportability.

"now i'm not saying that we need to go back to the 90s and early 2000s and you know create these these waterfall and massive gantt charts or anything but what we should be doing is taking the time upfront to be able to not just define what is this individual thing that we are building but where does this fit into the overall picture so so at least do that so that we're not doing things like duplicating functionality that another team may have already built is this something that could be reused great that is time well spent if it takes you one and a half times to build this in an extensible way then that is better than the two x time from a business perspective that it would have taken to build it twice"

Totenberg advocates for a return to more intentional planning, without reverting to old waterfall methods. He emphasizes that taking time upfront to understand how an individual feature fits into the larger picture prevents duplicated effort and promotes reusability. Totenberg argues that building something extensibly, even if it takes slightly longer initially, is more efficient in the long run than building it twice.

"so for each of these different categories of releases we should be able to have not just the control strategy but the measurement strategy right so how do you actually know how will you confirm that this thing is going well and there are likely some pretty good reusable metrics this is actually one of the reasons that i was talking about a centralized observability strategy before is like if there's high level things that you care about is the overall system actually up or not which is you know we've got status pages and and we should be able to see if that microservice is responsive or not that sort of thing we should be able to reuse these but but they're not going to be the same signals for every single change category right"

Totenberg stresses the importance of both control and measurement strategies for different release categories. He explains that reusable metrics, often facilitated by a centralized observability strategy, are crucial for confirming that a release is performing well. Totenberg notes that while high-level system status is important, specific signals will vary depending on the nature of the change being released.