A long time ago I wrote a fiction novel. 120,000 words whittled down from about 175k. It turns out it wasn’t any good although looking at it now some 20 years later it does have its moments. A nice turn of phrase here, an interesting description there. Although it was never published it was written and stands complete. For a week I outlined the novel, sometimes working on chunks then arranging those into a puzzle with pieces missing. I then added scenes to link these chunks together to create a narrative that I thought made sense. After another week or so of arranging the outline, I sat down and every day wrote 2,500-4,000 words, starting at one in the outline and ending at the next. By following the outline and writing from one element to the next, focusing only on the goals laid out in the outline while avoiding detours caused by tangents that weren’t relevant to the plot or the characters, within eight weeks I had completed a rough draft of the novel. I then spent the next four years editing and revising it, reviewing and rereading and re-everything , doing anything I could think of to make the novel shine. But it never did. It was still terrible. Hackneyed and predictable plot. Unbelievable characters who would be complimented by being called “two dimensional.”
Fast forward two decades and I’ve achieved my dream of being a paid writer. Sort of. As a systems analyst in the financial industry I am paid to write requirements documents and detailed software specifications. I have put together specs longer than my novel that could be measured by their thickness in inches if anyone dared print them out (people stopped doing that about 10 years ag0.) I have also put together specs that could fit into a PowerPoint presentation with enough space for goofy stick figure clip art. What differentiates the two is not my writing skills or even the size of the project: it’s the software methodology used by the institution.
Basic software design follows this process: People get together and decide on a solution to a problem they have and create a set of business objectives. A typical business objective that I deal with might be, “Let’s cut down the time it takes to report on delinquent accounts to senior management.” These objectives then determine the business requirements (the “what” of the project) which determine the functional requirements (“how” the business requirements are achieved), followed by the detailed design specs which tell the developers and coders what they need to build. The coders then code following the design spec and afterward conduct basic tests on their code to make sure it functions. The testers then work backwards, creating a test plan based on the functional requirements, then actually test what has been coded to make sure what the developers and coders coded actually matches what was laid out in the functional requirements specifications. Wrap the whole thing in a traceability matrix that ties the project objectives to the business requirements to the functional requirements to the tech specs to the testing documents, add in issues tracking for the inevitable bugs found and corrected before rollout, and you have a software project.
In software design there are two fundamental methodologies: “waterfall” and “iterative.” Waterfall methodology uses the metaphor of a series of waterfalls with one waterfall feeding another downstream. This requires all the project objectives to be clearly defined at the beginning of the project, the “waterfall top.” It assumes that you know everything there is to no about your business environment and needs up-front. The objectives cascade down to the business analysts who develop the business requirements before passing the documentation to the systems analysts, who produce the functional specs. Each team member does his or her assigned task without input from those who created the documentation “up stream” and is not involved in the consumption of the spec s/he creates by “downstream” developers, coders and testers. Once you produce your delivery artifact, the requirements document or functional spec for example, your role on that project is complete and the documents you created are expected not to change.
The iterative methodology starts with the business objectives, but instead of defining them all so that they can be codified into requirements, the expectation is that they will change and be added to throughout the process. In contrast to the waterfall methodology, the expectation is built into the process that you do not know everything about a particular system or business process at the project’s beginning, and you will learn as you go along. Documentation for these types of projects tend to be brief with lots of edits and versioning.
There are several different types within each methodology. Common iterative approaches are “Agile“, the first true iterative methodology developed in the early 1970s and “Extreme Programming,” developed in the 1990s but based on lessons learned during the Apollo space program. Some try to combine aspects of both methodologies. For example Scrum, an iterative methodology, takes what I consider a more waterfall approach by breaking up business objectives and spreading them throughout a project. This provides a more flexible approach to meeting a particular business requirement without changing the business objectives set at the project beginning which do not change through the project.
Most software projects fail. The reasons for these failures depend on who you talk to. As an analyst I often blame poor requirements documentation and questionable analytical techniques as well as spaghetti coding by developers who never invested time in reading the requirements and testers who were more concerned about ticking off check boxes than they were in actually using their brains and finding errors. But by far the greatest source of project failure is upstream with the decisions made by the business at the project’s inception.
What got me thinking about all this was an excellent piece by Clay Shirky on the failure of the Obamacare website. He cites Waterfall methodology. “The preferred method for implementing large technology projects in Washington is to write the plans up front, break them into increasingly detailed specifications, then build what the specifications call for. It’s often called the waterfall method, because on a timeline the project cascades from planning, at the top left of the chart, down to implementation, on the bottom right.”
Waterfall methodology has its place, although where that place is eludes me right now. The problem I have with waterfall is that it’s great for simple projects with a small set of clearly definable project goals and requirements. But complexity demands too much from the methodology which is why I find its pure form so rarely used in design these days. Most projects I’m involved are huge project impacting numerous business lines, data warehouses, and outside vendors. It is impossible for management to know all there is to know about their own business processes and systems, and the smart managers don’t even try. They speak in very broad, general terms and leave the impacted technical teams to hash out the details. That “hashing out” usually requires in depth analysis and reverse-engineering of the impacted systems designed by developer no longer with the institution from poorly detailed and written specs that were stored on someone’s hard drive that got wiped once they quit.
Shirky continues, writing, “By putting the most serious planning at the beginning, with subsequent work derived from the plan, the waterfall method amounts to a pledge by all parties not to learn anything while doing the actual work. Instead, waterfall insists that the participants will understand best how things should work before accumulating any real-world experience, and that planners will always know more than workers.”
This is a particular conceit of the Obama administration and bureaucrats in particular. One of my core beliefs is that the Law should leave a “light footprint” on a free society. It is impossible for legislators to write laws that are capable of responding to every circumstance, therefore laws should be written carefully to give the citizenry the benefit of the doubt, and give prosecutors and judges latitude to decide violations of the law on a case-by-case basis. It’s one reason why I oppose mandatory sentencing rules and making abortion illegal even though I recognize it as murder. Unfortunately legislators and bureaucrats don’t see their job that way. They strive to make new laws and write new regulations instead of making those that exist more effective and less onerous on the citizenry.
In the case of Obamacare the Obama administration thought it understood how to design software. It is a typical show of arrogance coming from the administration who brought us the “Reset with Russia” resulting in a new Cold War, supported the Arab Spring which has resulted in everyone in Egypt hating America instead of the two-thirds of Egyptians who hated us prior to the Obama administration, and now the Iranian Nuke Deal which results in Iran acquiring nuclear weapons. Giving this administration power was like giving hookers, cocaine, cars and guns to a group of teenagers. It’s going to take decades to undo the damage this administration has caused.
But in the meantime we have Obamacare. As the one lemming said to the others, “Forward!”