Very few automators have the time to automate tests to cover every test case for every feature. However, it is usually possible to either automate most test cases for SOME features, or SOME test cases of EVERY feature. For this reason, it is a good idea to consider from the outset what the primary goal is for your automation.
1. Subsidise manual testing effort (Automate most test cases for some features). The idea is to run automated tests in place of executing them manually, thus freeing time for the testers to primarily focus on testing newer and more complex features. Ideally, in this situation, the aim is to automate as many tests as possible, without necessarily focusing on feature coverage. In this way, “old” features can be almost fully automated, and require minimal manual testing, leaving the testers more time to explore new features.
2. Automatic Build Verification (Automate some test cases for every feature). The idea of this one is to have a suite of automated tests that are light on complexity, but rich in feature coverage. This kind of automated test suite is designed to run a “sanity check” on the new build to make sure that the most critical components of each feature are still functioning as expected.
Gratuitous analogy about cows
Say we are testing a cow. A cow has a number of different features:
1. Eats grass (9 test cases defined)
2. Produces milk (4 test cases defined)
3. Moos (16 test cases defined)
4. Looks like a cow (22 test cases defined)
Because of code reuse, let us estimate the following (this is an example estimate and won’t be true for all projects of course):
Time to write a script for an unautomated feature: 2 days
Time to write a script for an automated feature: 0.5 days
Farmer Beans has 6 days to get some automated testing happening on this cow. So he can choose from the following:
1. Automate the scripts to check that the cow eats grass, and manual test the rest:
Time management: 2 days for the first script, and 4 days to write 8 more scripts.
This is a good option for Farmer Beans if the cow was already programmed to eat grass in the first iteration of his project, and the developers are currently working on how to make it produce milk, moo and look like a cow. It will free up Farmer Beans’ time to test the new features without having to constantly manual test the same tests to check that the cow is eating grass. It also means that if there are any changes in the new features currently under development, that his automated scripts are unlikely to be affected and so it will result in less maintenance time.
2. Automate one high priority script from each feature to run against each build
Time management: 6 days to write 3 scripts
This is a good option for Farmer Beans if there all areas of the cow are going through some frequent changes. Perhaps there are many small change requests to change the way the cow looks and functions. In this case, Farmer Beans will want a quick way to make sure that the most critical functions of the cow are staying the same, so that he doesn’t have to spend a lot of time maintaining his test scripts and can focus his time on testing all of these frequent changes. In addition, with all these changes happening, Farmer Beans will be quite worried about the stability of the build on a daily basis and this will provide him with a quick way to let him know if the application is broken without him having to read lots and lots of relatively minor error messages.
Long term goals
In the short term, you will only be able to meet one of these goals. However in the long term you may be able to meet both goals. For example with Goal 1, theoretically there will come a point where most of the test cases for most features are automated and at this point it could be worth defining a subset of these test cases as build verification scripts and prioritising test cases to make up the missing feature coverage. Similarly with Goal 2, once there is a suite of test cases testing all features, it should be easy to reuse the code from each of these in order to automate more tests for every feature.
Of course this is providing the situation allows for it. Lack of testing resources and frequent application changes will always reduce the chances of meeting both goals, so you may just have to stick to one goal so that you don’t go crazy trying to meet them both. Remember, automation is meant to make testing more efficient. If it’s slowing you down, rethink your automated testing strategy – you may be trying to do too much with the resources you have.