Explaining different types of testing

I was recently involved in a software release where some rather unfortunate defects made their way to production. This resulted in some (not unreasonable) tough questioning from our stakeholders.

One particular area they wanted to ask about was our testing practices. Were we not testing our product the same way their user community used it?

This highlighted an important disconnect in understanding the variety of types of testing that a responsible, professional team would usually employ to thoroughly test a product.

It was true that our testing wasn’t good enough for that release. But there was much more to it than simply “testing the software in the way users use it.”

In a bid to help them understand our weak areas (and what we were doing to remedy those) I went hunting for one of those “test pyramids” modeled on the same idea as the USDA food guide pyramid (http://www.nal.usda.gov/fnic/Fpyr/pmap.htm)

There are plenty out there, but I wanted to assemble my own which is what you can see here (click to enlarge):

If I were to do this again I’d probably separate out a bit more the different types of testing in the lower blue section for better emphasis of each, but you can get the gist of things from it as is.

As you can see, I’ve classified the testing appropriate for our product in quite a few different ways. Understanding what these all were, how they differed and why each different kind was necessary was (it became apparent) difficult for our non-technical audience to easily understand.

Reflecting upon this later, I came up with the following analogies that I hope helps make these different kinds of test more understandable to non-developers (and even some developers...)

Why so many different kinds of test?

This too was a puzzle for some. The idea I came  up with to explain this was that, just as a doctor may need a wide variety of tests to diagnose a patient, so too we need a variety of tests to check the "health" of our software.

The analogies below are all based around the idea of building cars. They might be a bet stretched, but I hope they're useful.

Unit tests

Inspecting each component in isolation. Confirm the building blocks are fit for use in the bigger system (car).

Feature/acceptance tests

Checking all features behave as expected, e.g. does it steer properly, doors and trunk open properly, seats adjust etc. 

Performance tests

Driving around a racetrack, time trials, checking how fast you can get from 0-60?

Stability and reliability tests

Driving the car non-stop until you've done 200,000 miles+ without (too many/too severe) problems

System tests

Driving around the neighborhood, highways, driving with passengers, kids etc. (i.e. using it the way real people would do)

User acceptance testing (UAT)

Getting the public to try it.

Exploratory testing

Driving around town, down the highways, mountain roads, 4WD roads, dirt roads, etc. You hope to find unexpected things.

Smoke testing

Giving each car that comes off the production line a quick drive.