Getting started with test automation

Test automation seems to inspire mixed reactions in the testing community. There are those who believe it’s absolutely essential, and those who shun it completely, with all shades of grey in between.

Are we all even talking about the same thing?

“Test automation” is generally used to describe any form of testing where some form of software is executing tests rather than a human being. This kind of testing comes in many flavours, such as:

– Scripted unit tests

– Tools that allow testing of services

– GUI-level test automation using record and playback tools

– GUI-level test automation using hand-coded frameworks

When I talk about automation, I generally talk about GUI-level test automation using hand-coded frameworks.

Why do some people mistrust automation so much?

Your own experiences with test automation will shape the way you see it. Here’s a pretty typical story that I’ve heard many times:

Once upon a time there was a test team that had thousands of manual test scripts to run and it took them a very long time to do it. One day, a friendly automated test tool vendor told them that they could reduce the time to do it if they just used a test automation tool to do it for them. The test team bought the tool and automated a large portion of their tests. After some time, they found that they were spending so much time fixing the automated tests that it really didn’t save them any time at all. What’s worse is that they weren’t finding as many bugs now that they were using the tool to test for them. But by then it was too late – they had already paid for the tool so they were stuck with it and had to use it.

Why do you like automation?

Here’s my story:

I started out as a developer, but my boss soon noticed I had much more of an aptitude for finding bugs than fixing them. So he encouraged me to try testing. So I joined a company that was big on test automation. With my developer background, I easily picked up automated testing. This company tested projects in the following way:

– Manual test scripts were created from requirements

– 80% of these test scripts were automated, using GUI-level test automation with hand-coded frameworks

– Automated tests were run every night. Any scripts not automated were run manually during regression testing.

And it WORKED. Projects were built in an environment with very strict processes and change procedures, with staff dedicated to test automation. The developers were all very experienced and built the software in such a way that it would be easier to automate tests for it while it was being built. The software was built to a consistently high standard and deadlines were met with this approach in place.

So that was an approach that worked FOR THAT CONTEXT. I saw this same approach attempted in a different context and it did not work at all. Without the strict change processes in place and the experienced developers building software with automation in mind, the testers were struggling to meet their 80% automation target. More time was spent fixing automation than looking for bugs. Bugs were missed, deadlines were missed – bad news all around.

So GUI-level automation built on a large scale can work. I’ve seen it work. It can also fail horribly. It’s just a tool. Understanding what it is and how it works will help you to know if it’s able to help you any way.

So should I use it or not?

Here’s my advice to anyone wanting to get started with a bit of automation:

1. Find someone who already knows about it and consult them. There is a massive learning curve involved with this kind of automation. Sure you can learn all about how to use the tools just from books and online information, but the trickiest part of using automation is knowing what the best approach is for the context of your project. That sort of knowledge is best gained through experience. If nobody on your team has that experience, it’s very costly to just try to implement one approach, and then use trial and error until you find the best one. It can cost you heaps of money in tools and training, and heaps more time in trying to get the damn things to work.

2. Know what you’re getting yourself into. If you do attempt to build GUI-level automation on a large scale, you are basically building another software application to test your software application. As testers, we already know how hard it is to build quality software. So it’s a big job, and it needs a lot of work, and some developer-level coding skills.

3. If you can’t go big, then think small. The biggest time waster with GUI-level automation is the maintenance. So have a think about if you can use GUI-level automation tools to your advantage without the maintenance cost. If you can train your team to easily script use-once automated tests that might speed up repetitive tasks, then maybe that’s worth looking into. Maybe you can use it to simulate constant user activity, or to generate and input large amounts of data. Maybe you can just automate a small suite, like a smoke test, which wouldn’t require much maintenance.

4. Don’t get suckered into using expensive tools. I’ve used expensive automation tools and free automation tools and quite honestly, the quality was about the same. Don’t get stuck in a situation where you’ve paid so much for something that you can’t justify abandoning it if it turns out to be a bad idea.

5. Remember, it’s just software, and software is buggy. We know this – we’re testers! So don’t rely solely on automated tests to tell you if your application under test is working fine. Use the automated tests results as a bug trap, but remember to check the trap in case it has a big hole in it. Use it as an aid, but never a substitute, for manual testing.

For anyone wishing to know more about automated testing, I just discovered The Automated Testing Institute, which seems to have a fair bit of useful information on the topic. Probably not a bad place to start. But definitely find someone to consult!