The Lurking Danger of Technical Debt

Em Blog Technical Debt Main Image

Software modernization is expensive, but not modernizing can cost even more

“If it ain’t broke, don’t fix it.” Sometimes it’s the right move, but applying that adage to software can potentially result in disaster when it finally does break.

Many companies are reticent to upgrade legacy software due to the cost or inconvenience. Unfortunately, that’s how you end up with government and financial systems running on COBOL, a software language first used in the 1950s. Salaries for COBOL programmers saw one of the biggest increases in 2022, according to a Stack Overflow survey, while only 0.65% of the developers surveyed are using it.1 It’s a skillset that’s becoming harder and harder to find, but it’s necessary to maintain aging mainframes.

Legacy software is a source of technical debt—making tradeoffs now that you’ll need to pay for later in the form of changes, upgrades, or modernization.

Technical debt was a major contributor to Southwest Airlines cancelling over 15,000 flights in late December. For the past year, the flight attendants union had expressed concerns about the airline’s antiquated crew scheduling software, because when the software fails, manual processes are required to locate and schedule flight crews.2 When a major storm cancelled flights, it caused a cascading failure that the software couldn’t handle.

“Our technology struggled to align our resources due to the magnitude and scale of the disruptions. As a result, our crew schedulers tackled the issue manually, which is a tedious, long process that takes time and trained resources to accomplish.”
-Chris Perry, Southwest Airlines, as quoted in The New York Times3

The perfect storm of holiday travel and bad weather created a nightmare for the scheduling team and illustrates the danger posed by ignoring technical debt. Leaders are entirely aware of the issues—a 2022 survey found that 41% of IT leaders and enterprise architects consider reducing technical debt and modernizing legacy systems the top IT priority.4 But fixing technical debt can be daunting and expensive.

McKinsey analyzed 220 companies to find that businesses with more technical debt struggled to prioritize the right projects for modernization, resulting in budget spend that did not fix the real issues.5 They also spent 50% less on modernization projects overall and were more likely to have modernization projects that failed.

Remember the Y2K bug? Older software only used two digits to indicate the year to save precious storage, meaning that many computers were going to think they traveled back in time on January 1, 2000. Y2K is a great example of technical debt that didn’t end in disaster specifically because of modernization efforts. An estimated $100 billion was spent in the U.S. between 1995 and 2001 to mitigate the problem.6 That’s nearly $179 billion today!7 Because the efforts were successful, it had the unfortunate side effect of making people think it wasn’t a big deal.

While it’s difficult to justify major spending to fix something that isn’t currently broken, McKinsey also found strong correlation between technical debt and business performance. Those with the least technical debt had 20% higher revenue growth than companies with the most debt.8

While bad weather was a catalyst for the cascading failures at Southwest Airlines, and technical debt doesn’t bear all the blame, the result is still expected to be very expensive. Bank of America’s airline stock analyst said the spate of Christmas week cancellations could cost Southwest between $600-700 million, including lost revenue from refunds and passenger reimbursements.9

Hopefully this incident serves as a motivator to those organizations still using software developed before many of their employees were born. While reducing technical debt is a costly endeavor, the risk mitigation alone may justify the effort.

  1. Stack Overflow, 2022 Developer Survey, May 2022
  2. The New York Times, The Shameful Open Secret Behind Southwest’s Failure, December 2022
  3. ibid.
  4. LeanIX, Enterprise
  5. Architects on IT Priorities for 2022, April 2022
  6. McKinsey, Demystifying digital dark matter, June 2022
  7. Washington Post, Y2K Repair Bill: $100 Billion, November 1999
  8. CPI Inflation Calculator, accessed January 2023
  9. McKinsey, Demystifying digital dark matter, June 2022
  10. CNBC, Southwest Airlines’ schedule stabilizes after holiday meltdown but costs are still piling up, January 2023