“Just test everything”

I get a little irritated when I get asked to “just test everything” and developers don’t understand why. So here’s why.

Let’s say we’re testing Twitter. To make it even simpler, we’re not testing all of it, just the web page where you type your thoughts into the box and click update. In fact, forget about the other friends, forget about the rest of the page, just focus on the box. How many test cases do you see there?

I see a few hundred, maybe a few thousand? It’s been a while since I did factorials.

Okay, so first there’s the obvious – entering valid data, making sure the number of characters updates, making sure the tweet is successful, blank values, more than 140 characters, WAY more than 140 characters, special characters, foreign characters, checking the layout is correct, etc. I guess there’s about 20-30 tests right there.

But then you have to remember we’re on the WEB, baby. So you can execute every one of those tests in a different web browser. And don’t forget versions. So there’s Firefoxes 2 and 3, Internet Explorers 6, 7 and 8, Safari, Chrome, Opera, Lynx…what’s that? You don’t think many people will be using Twitter via Lynx, the text-only browser? I’m sorry I must have misheard you, I thought you asked me to test everything.

And hey, while we’re talking web browsers, what about different operating systems? Each of those browsers has a different version to operate on a different operating system, so there’s a slight chance that they may display things differently on each one. Take a minute to think of how many combinations that creates. Plus different screen resolutions, and what about different brightness settings? Dark text on dark backgrounds might look sexy on your ultra bright monitor, but what about those who prefer dimmer lighting?

I haven’t even started with security – SQL injection, cross-site scripting and all that jazz. Then there’s performance – how long will it take to post your tweet? What if I write 100 tweets, will it still be fast? What if I write 1000 tweets? Did someone mention accessibility? Is the button an image, and does it have alt text?

Ah now you think I’m just being picky. Okay so, let’s take a look at that innocent-looking little piece of text under the box that says “about 1 hour ago”. So there’s a test to check if it measures in minutes until it gets to 1 hour, then a test to check if it measures in hours before it gets to 1 day, then to test if it measures in days before it gets to 1 week, and so on. I’m not sure when that thing stops actually. Could we be testing this a year from now? Now think of all of those tests, and then think…what if I write a second tweet? Will the number update? And will the unit of time update as well? Have I checked this for all numbers and all units of time? Now what if I write a third tweet?

Okay so all of those test cases for one little box. So how many features do you have in the software you’re developing? How many of those features are connected to the other features? How many test cases do you think there might be?

Can you see why I don’t test everything? I’d keep going forever!