Saturday, January 19, 2008

Book review: "Dreaming in code"

If I were to describe this book to someone in one sentence, I would say that this book explores the philosophy of software development. I guess this topic is so much out of the ordinary that it is only useful and interesting to people whose lives revolve around producing software. And yet this isn't even a technical book, but a fairly successful attempt at fiction. Anyone should be able to read it, although people with little technical savvy may have to look up a meaning of a word or two here and there. The author, Scott Rosenberg, is not a famous programmer himself, but he is a co-founder of Salon magazine and has been around famous software projects for many many years, and is well qualified to write about the subject.

He sets out to explore a deceptively simple question: why is software so hard to write? Why is so much of it so bad, and why is it never finished on time? Why is everyone, both users and developers, always complaining about software? Essentially, why can't we build software like we build cars, like we build bridges?
He decides to explore the topic by joining a moderately large software project (Chandler) as an independent observer. He uses the story of this project it as a skeleton around which he builds his book by discussing all the innumerable troubles that befall almost all software projects.

The approach works, but sometimes Chandler gets too much in the way. Details of how project evolved really take too much space in the book, but thankfully you can get over those parts quickly and dive straight into the juicy parts. And to me the juiciest parts were where other (in)famous software projects are mentioned and discussed. The books is full of little anecdotes, stories and quotes, starting from the humble beginnings of Computer Science and ending in present. And this is really where the book shines. The author really did his homework and went in every damn dusty corner of software development history looking for answers. You will see quotes from Turing, Brooks and, of course, Knuth. You will hear many stories of software disasters of epic proportions, of project failures that are so tragic they rival Babylonian Tower itself. Obviously, Rosenberg sprinkles his own conclusions and thought all over that, but in a really non-intrusive way. He acknowledges that he doesn't really have answers and you are always free to draw your own conclusions from the stories he tells. I appreciated that attitude; and I did draw my own conclusions which were actually different from the ones author made.

The book ends by reluctantly admitting that we don't really know why we keep screwing up our software, maybe because the reasons are many, or maybe because we just haven't found that magic recipe for building perfect programs.

What is weird, is that I felt exactly the same - but before I read the book. However, once I was done I had a completely different feeling - I felt that I actually understood something and could now answer the book's question. I do not have a recipe for baking perfect code, but I think I kind of can explain now why software can be so bad and ugly.

The epiphany came (and I hope the author had one too) when I was already reading the epilogue. In it, Rosenberg suddenly comes back to the questions he set out to answer, and especially "Why can't we build software like we build bridges?". As if he has given up on finding the answer in the history of software, he suddenly asks: "And how, really, do we build bridges?" And that is where it becomes obvious that when we build bridges we mostly suck, too. Not only did bridges crash left and right for most of our history (only very recently did we learn to build them well, and even now there is a bridge fall every now and then) but there are plenty of examples where bridges could not be built on time and on schedule. So there really isn't anything special about software world. We are people, we are not perfect, and we are prone to screw up every project we work on, no matter what field. Maybe the only thing different about software is that it is so flexible and easy to change, it takes much less effort to mess it up.

So in the end, even though the author felt like he hasn't found the answer, I felt like I have. Here is my take on the question: "Why is software so hard?" It really breaks down into three questions:

1. Why is most code of such a bad quality?
The main problem with software is that it is too easy to change it. When you build a bridge, you know it will be very costly to change design mid-way. With software, people are tempted by the ease with which they can make changes. This leads to two traps: first one is in planning. People think that they can change anything they want later and start building a project without a firm idea of what it is. You know, like starting to build a bridge, but not knowing where the other side is, and if it exists at all. Of course in many cases that will lead you to spaghetti code. Another trap is enhancements. Once people have completed the original design (if they had one) they are tempted to keep "improving" it. After all, it is so easy to add features - a line of code here, a line there... This process (almost by definition!) continues until it is no longer easy to add new features - why? - because code has become a mess of extras. That is when projects get scrapped, recoded, replaced, etc. And everyone blames the old team who "couldn't program" and created a mess. But of course, the new team eventually ends up with the same quality code, after all the useful extra features.

2. Why are deadlines so often missed on software projects?
Well, simple. Programming is in large part a work of art and another part is invention. Writing software is not like building cars, it's like designing them. Ask any scientist how well they meet deadlines on their research - most don't even set deadlines! The thing is, we do build software better than cars - it's called copying an executable, and we do it very well. But designing software is not easy. Almost every software project is doing something that no one has ever done before - otherwise they would just copy the code! When you are doing something that nobody ever done before, how do you know how long is it going to take? Who knows.

3. Why is most software so buggy and under-developed?
See previous point. Most managers insist that software is shipped by certain date, no matter what. Since it isn't really possible to predict how much work will be done by that point (see above), a large fraction of projects go live in unfinished condition. In my experience, only companies that do not set deadlines on their work produce quality results. There are many examples of companies who just said "to hell with deadlines!" and they usually produced nearly perfect, very successful products. Blizzard Entertainment is one example.

As a conclusion, my opinion on this book is mixed. I do recommend to read it as a collection of extremely fascinating stories about the history of software development. But in all other respects, your mileage may vary - it may help you think about software development process, but it may or may not give you insights into how to improve it. You may also find all the Chandler-related stuff somewhat boring.


  1. A bunch of scientists I know would tell you that they meet publication deadlines by burning the midnight oil up to the deadline. But that clearly involves total focus on the problem of describing the contribution and findings, not in doing all of the work that lead up to the conclusions. In other words, you can set a deadline for documentation but not for the product.

  2. 1. Why is most code of such a bad quality?
    IMHO: Lack of coding guidelines, lack of design reviews (lack of planning), lack of code reviews, lack of tests. In summary, "what is the first thing that pops into my mind..." becomes the code.

    2. Why are deadlines so often missed on software projects?
    IMHO: Feature creep, lack of planning and design. Late focus on testings and refinement rather than on continuous evaluation. Setting a fixed date rather than a time-boxed date that is larger the further out it is set.

    3. Why is most software so buggy and under-developed?
    IMHO: customers don't buy the new version because it has fewer defects, they buy it for new features (or the unavailability of the old version). Waiting until the second version/service pack/etc is the only controls customers have on specifying "as fewest defects as possible."

    See Steve McConnell's "Software Development's Low Hanging Fruit"

    And IBM's best practices:

    "Most software projects fail. In fact, the Standish group reports that over 80% of projects are unsuccessful either because they are over budget, late, missing function, or a combination. Moreover, 30% of software projects are so poorly executed that they are canceled before completion. In our experience, software projects using modern technologies such as Java, J2EE, XML, and Web Services are no exception to this rule.

    This article contains a summary of best practices for software development projects. Industry luminaries such as Scott Ambler, Martin Fowler, Steve McConnell, and Karl Wiegers have documented many of these best practices on the Internet and they are referenced in this article. See also the Related information section at the end of this article. The companion article, Guide to Running Software Development Projects, describes the top ten factors that help improve the success of your project."