ArticleS. UncleBob.
UntestedCodeDarkMatter [add child]

Untested Code is the Dark Matter of Software

(Apparently because it makes up 90% of the software universe)

In a recent blog Cedric Beust quoted a posting of mine from the junit mailing list to the effect that our code should have test coverage in the high 90%s. To ship code without such test coverage is unprofessional. Cedric's response was:

That's a bit extreme, but not entirely untrue. What this statement fails to distinguish is that there are several levels of "unprofessionalism". I can think of a few that are way more critical than "shipping code that it not covered by tests":
* Missing the deadline.
* Shipping code that doesn't implement everything that was asked of you.
* Not shipping.

The point being that the only way to make the deadline and ship is to KNOW that you are not shipping shit. And that requires high test coverage.

Now, you can play the game. You can gamble that your code works even though you haven't tested it thoroughly. And lots of times you'll get away with it. But it's unprofesssional behavior if you ask me.

Cedric goes on to say:

There are plenty of ways to know that your code works. Testing it is one. Having thousands of customers over several years, consecutive successful releases and very few bug reports on your core functionality is another. Claiming that only testing or code coverage will tell you for sure that your code works is preposterous.

I'm not sure why Cedric thinks that having thousands of customers test your code for you is professional. The fact that the code generated very few bugs is good, and speaks well of the people who wrote that code. But if the code was shipped to customers without a high degree of test coverage, then the programmers took a significant risk, which is unprofessional.

Finally Cedric claims that there is a middle ground between tested code that you know works, and untested code that you don't know works:

There is actually something in the middle: it's called "Code that is not tested but that works".

I agree that it is possible for untested code to work. However, it is not possible to know that the code works unless it is tested. And, indeed, your knowledge of whether the code works or not is related to the amount of test coverage you have.

Now I'm sure that Cedric tests his code to some extent, perhaps a lot, perhaps a whole lot. I hope so. What concerns me is the notion that shipping take precedence over testing. That deadlines are more important than quality. That it is better to trade features for quality.

And finally, the thing that bothers me the most, is the idea that high test coverage is somehow mutually exclusive with shipping on time with a full feature set. It seems to me that the best way to ship on time with a full feature set is to do whatever you can to eliminate debugging and "stabilization" time. The way to eliminate the vast proportion of that time is to write your tests, and to write them first. The best way to prevent debug waste is to never leave the code uncovered.

!commentForm -r
 Fri, 20 Oct 2006 19:44:10, Mike Bria, The Next Guy
There is one very important benefit of good test coverage that I'm a bit surprised to see missing from this chain - protection from/for "the next guy". Meaning this: even if /you/ are a genetically-altered coder who can naturally write 100% bug-free code 100% of the time, without tests your perfect code is at high risk to be speedily turned to bug-ridden mud by the poor soul who inherits your legacy once you've moved onto management.
 Tue, 26 Sep 2006 17:34:42, Danny, nitpick
Although I agree with your general sentiments, "it is not possible to know that the code works unless it is tested" is ignoring the points (effectively made in previous comments) that 1) if the app works at all, it's passing some tests by implication; 2) having lots of tests doesn't mean the code will work under all conditions. It may be helpful if you distinguish between broad high-level tests (it appears to work overall), finer-grained unit testing (these parts appear to work in isolation) and correctness (it *should* work).
 Wed, 6 Sep 2006 21:11:09, Lionel Orellana, No tests at al or manual tests

I wonder if "Code that is not tested but works" actually means "code that is manually tested but works". I find it very hard to believe that anyone would write any program without at least trying to use it and throw some bad input at it to see what happens. That is testing too and I'm sure we all do it regardless of deadlines.

I could buy the argument that manual testing can ensure the same level of quality than automated testing (albeit with less efficiency) and I could agree that automated tests are not the only way to tell your code works. Sure you can allocate a bunch of resources to do manual testing and do it all over again when anything changes. Sure you could even reach the same level of coverage and ship quality code (even though you couldn't actually measure coverage).

The argument is that automated tests save you time compared to manual testing if only because you can do regression testing with one click.

The time it takes to actually write the tests depends on how familiar you are with the tools you are using. There will be a learnig curve but after that the tool becomes seconde nature and writting the test is just part of the development task. You have not completed a task until you have a test for it (something TDD takes care of nicely).

In this way testing is no longer this extra burden at the end of a cycle (be it manual or automated tests) but it is an integral part of development itself.
 Tue, 5 Sep 2006 22:32:29, Aaron, Cedric's point is too idealist
I think Cedric's point is good, I mean the middle ground may do exist, but his point his not good enough. There may be some guys who can write code without any testing, while they can also reach the quality that the test coverage over 90% could have. I am really admiring those guys who can write that sort of beautiful code, but the point is how can you identify who are that sort of guys, and how can you prove this right guy can write such a high quality code at this right time. I really doubt about that.

And somewhat this is the same as the US law system, you have the right to keep silent, but if you have something to say to the court, you have to get the evidence to prove it first.

 Tue, 5 Sep 2006 17:02:53, George Dinwiddie, some of the responses are amazing
There are many (including Aaron Korver, below) who point out that 100% code coverage doesn't imply correctness. Of course, Uncle Bob never claimed that; just that low coverage implies ignorance about correctness.

Some of Cedric's supporters on his blog posting really amaze me, though, and I wonder if Cedric is embarrassed to have them as supporters. Some mistake code coverage with all possible permutations of input: "There is no and never will be any code that is 100% tested. The time and effort to provide 100% testing would require infinite resources and time" Some seem proud of shipping buggy code: "we had extrem [sic] deadline, so we programmed it, shipped and then tested. there were lot of bugs. But having tight deadline that was our only alternative -- and we succeded. By the way, that software is for military logicstics use, and is installed in several countries:-) And yes, around 80% of code is NOT TESTED and it works without critical bugs." [I wonder if these critical bugs will only show themselves at critical times.] Some imply that testing is antithetical to real work: "This is why overzealous consultants who have never worked on a product with multiple releases and stiff competition have no business advising others what they 'should' be doing. It's almost downright reckless." Do they think that Cedric is arguing for lower code coverage on principal?

All, of course, assume that tests with high code coverage take a lot more time. Personally, I find that TDD development *saves* me time, and gives me a pretty respectable test-coverage by itself. Unit testing is not a panacea, but it's certainly no slouch at improving quality (when combined with corrective action).
 Tue, 5 Sep 2006 16:06:52, Aaron Korver, One last quote for the road
Be careful about using the following code -- I've only proven that it works, I haven't tested it.

Donald Knuth
 Tue, 5 Sep 2006 16:00:02, Aaron Korver, Coverage = Quality?
I'm sorry, but how does having a high test coverage mean that we aren't shipping shit? I defer to Steve McConnell[?].

"Testing by itself does not improve software quality. Test results [and code coverage] are an indicator of quality, but in and of themselves, they don't improve it. Trying to improve software quality by increasing the amount of testing is like trying to lose weight by weighing yourself more often. What you eat before you step onto the scale determines how much you will weigh, and the software development techniques you use determine how many errors testing will find. If you want to lose weight, don't buy a new scale; change your diet. If you want to improve your software, don't test more; develop better."

Steve McConnell[?] Code Complete

Thus, lets focus our attention on the development practices (TDD included) that help the quality go up, and stop the silly debate about code coverage.
 Mon, 4 Sep 2006 04:08:16, Steve Rees, the Pereto Principle
The cost of going from 80% coverage to 100% coverage may well follow the Pereto principle if you write your tests after you have written the code and write the code after you have completed your design (ie. classic waterfall development). However, you have probably spent a lot of time writing code that isn't used or introducing unnecessary flexibility into your designs.

Following a more agile approach that is based on Test Driven Development where the tests are written before the code and the design is continuously refactored in light of the changing needs of the project can dramatically affect the amount of code that is written, and that therefore needs to be tested.

This goes a long way to paying for the "extra" 20% of test coverage. The rest will be paid back in reduced debugging and fixing of the software once it has been released.
 Sun, 3 Sep 2006 20:19:01, Geoff Glasson, RE: Untested Code is the Dark Matter of Software
In my opinion, our view of professionalism defines how we approach software development as a whole. If our view is that "I will do my utmost to deliver well designed, clean, and tested software" then we will naturally gravitate towards thoroughly testing the software we write. On the otherhand, if our view is "Ship it at any cost" then we won't. Unfortunately it's not as black and white as that in practice because of the variety of road blocks put in the path of a software team.

As a Project Manager who started out as a Software Engineer, I have come to realise that we must find an appropriate balance between delivering well tested code, and delivering less well tested code. Note that I said "less well tested" because we should NEVER ship untested code since it will come back to bite us. In my view, the goal should always be to develop tests that thoroughly test your code. Unfortunately, in my experience the cost of going from 80% coverage to 100% coverage follows the Pereto Principle (80:20 rule) and is therefore very expensive. I'm not convinced that we can get to 100% code and branch coverage in all cases, but we should endeavour to cover as mush as possible - preferrably > 80%. Sound engineering judgement must be applied when deciding on the level of testing being performed. I don't believe that a it is unprofessional to stop testing at 80% coverage if there are valid reasons to do so; however stopping at 80% WITHOUT having a sound reason IS unprofessional in my view. In the absence of sound reasons, we should strive to get the highest coverage we can.

A previous post hit the nail on the head (unfortunately) with the statements "most of the companies are in this business because of the money. Shipping shit so that nobody notices is a good business if you already got paid". This is poor business practice in my view, however it seems to be the predominent view in industry and it reflects badly on our profession. Regardless of whether we work in a team dedicated to shipping quality software, it is my belief that each Software Engineer must choose whether they are going to thoroughly test their code or not. A professional Software Engineer will use their engineering judgement when making that decision. Professionalism is a state of mind, and can be applied in any organisation.
 Sun, 3 Sep 2006 18:22:49, Dean Wampler, How Do Language Features Help or Inhibit Code Test Coverage?
Andrew Wall's comment about the need to test branches in order to approach 100% coverage got me thinking about how language features, and the way we use them, affect our ability to test comprehensively. I'm working with a client which has used way too many massive case statements and if blocks where polymorphism would have been preferred. (See the reafactorings in Fowler's book, "Replace Type Code with Class", "... with State/Strategy", "... with Subclasses". It just occured to me that the preferred polymorphic solution is much easier to cover with tests; something that isn't always mentioned when discussing refactorings like these.

What about language features themselves? Besides polymorphism through inheritance, do features like closures and continuations improve or inhibit testability? Has anyone researched this subject?
 Sat, 2 Sep 2006 18:54:57, ,
Is being professional important?

I mean, most of the companies are in this business because of the money. Shipping shit so that nobody notices is a good business if you already got paid.

That's what happens with most software projects and project managers know that. They might test critical code, meaning code that if it doesn't work, they may get sued. But otherwise, it makes no business sense.

I agree that it is unprofessional. On the other hand, companies are compared on ROI, not on professionalism. That's why most projects fail, by the way, because they guess wrong. The best professionals will never get the big bucks because they will walk the extra mile to make sure the software works before they ship. This means loyal customers and less ROI.
 Sat, 2 Sep 2006 06:02:50, Daniel, Citing
He names you in the second paragraph.
  • Yes, he send me email the day I posted this, and very courteously asked if I would prefer being cited, to which I answered in the affirmative. So the citing, and the courtesy issues have been resolved.
 Sat, 2 Sep 2006 05:38:16, Andrew Wall, Code Coverage
If you search the internet you will find comments about code coverage. Several people say that 100% coverage is a minimum to ensure quality (they mean that you also need good branch coverage).
I would love the code I work with to have automated tests and I am working to improve the situation and I use TDD when I can to add new features.
However, I can confirm that 100% line coverage /is/ possible and >98% is a natural outcome of TDD.