I’m not sure, but every time I review this story or tell it to others, it has me wonder if this isn’t one of the biggest scandals in the history of productivity and process.

Based on what I’ve read in Agile and Iterative Development – A Manager’s Guide by Craig Larman, repeated in Agile Data Warehousing by Ralph Hughes and apparently confirmed in Wikipedia, it would seem that my unease with waterfall has some basis.

I’m going to quote Hughes directly (Hughes, p.48):

The vast majority of corporate IT departments today still utilize methods based upon the waterfall method. How can we foster the right frame of mind among IT managers that they will give Scrum/XP a honest try? We will draw upon two stratagems. First, point out that the predominance of the waterfall method is due to a tremendous error on the part of the U.S. military when it was codifying software development methods, and second, provide a long list as to why Scrum is many times more effective than a traditional approach. We deal with the military’s SNAFU here, and leave the advantages of Scrum for the next section.
The standard waterfall approach became the standard paradigm only through a series of mistakes. In 1970, W.W. Royce first identified the classic waterfall categories as only “essential steps common to all computer program developments, regardless of size or complexity.” In the second half of that same paper he clearly advocated an iterative path for progressing through these steps. To pursue the waterfall steps sequentially from analysis through coding in a single pass, he cautioned, “is risky and invites failure.” The single-pass waterfall method is only the simplest possibility for organizing a development project, and he suggested that “it only works for the most uncomplicated projects.” [Royce 1970]
Undaunted by these warnings from Royce, the Department of Defense proceeded to make the single-pass waterfall method its standard approach for systems development with its official specs MIL-STD-1521B and MIL-STD-2167. One can only speculate that they read just the first half of Royce’s paper and missed the later sections that contained his recommendations. The MIL-STD-2167 spec was particularly influential and soon propagated into many further standards both within the U.S. and internationally. Yet, when interviewed in the mid 1990s, the author of MIL-STD-2167 admitted that he had been unaware of the notion of time boxes when he drafted his recommendations, and with hindsight he would have strongly advocated iterative and incremental development methods instead. [Larman 2004 and Oestereich 2006]
In 1994, DOD replaced MIL-STD-2167 with an IID-promoting standard. Unfortunately, the other governments and standards bodies that based their methods on the earlier spec did not update their policies to match, so the single-pass waterfall approach sadly remains the basis of many methodological standards today. However, one can argue that to cling to a non-iterative, incremental approach is to persist in making a known, fundamental error in software engineering.

(Links to attributions mine.  “Oestereich 2006” attributed by Hughes as “www.isqi.org/isqi/deu/conf/conquest/2006/slides/B3.pdf (accessed Feb 2007)” but not available at the time of writing this blog post.)

I’ll be blogging more about my understanding of the history of agile, but the short version is that the failure of waterfall did not go unnoticed.  Overtime, folks realized that doing more of some things and doing less of others worked better, and that has lead to agile as we know it.