The trouble with GUI automation

It’s difficult

I think people have a tendency to greatly underestimate the difficulty of writing a good GUI-automation suite. It’s not as simple as record and playback, and it’s not like building regular software.

I’ve seen experienced developers underestimate it many times. Inevitably, they end up getting very frustrated and complaining about how rubbish the automation tools are. And yes, the poor quality of automation tools is definitely part of the problem. Developers are used to using very polished tools, lovingly crafted by developers, for developers. Testers are used to using either bloated overpriced commercial tools, condescendingly oversimplified so that “anyone” can use them, or well-meaning but under-maintained free tools, struggling to keep up with the latest technologies.

Let’s face it, the choice of tools isn’t great

Every time a new browser version or web technology hits the market, GUI automation tools suffer for it. Did any other test automators get a slight chill when they heard about HTML5? We only just recovered from the introduction of AJAX.

To be honest, I don’t think cross-browser GUI automation provides that much value anyway. Most cross-browser bugs are not the kinds of bugs that are likely to be detected by automation. The amount of effort that goes into to trying to get the automated tests to run cross-browser and cross-platform is usually not worth the amount of bugs found. If automated test tools were better, it would be a simple matter of changing a setting to make them all just work. But to my knowledge and experience, most if not all of them are just not there yet.

Experienced test automators write a framework on top of the out-of-the-box automation framework for all of their automation projects. Some people keep a generic copy of this framework that they can take from project to project. Why is this even necessary? Why isn’t it just built into the existing framework? Sure, everyone has their own special implementation, but the underlying principles are usually the same.

It’s still evolving

Then things like Project Sikuli come along. It uses image comparison to find controls in the app. As in, you can see pictures of buttons and stuff in the automation code. When I first heard about this, I thought it must be dodgy as all hell. Automated tests that use bitmap comparison to verify images have always been very unreliable in my experience. But I have been reliably informed that this does actually work. Is it a good idea? Honestly I have no idea. I have a hard time working out how this would be advantageous. But it is interesting because it is a more true version of GUI automation. It’s not just looking at the DOM anymore, it is actually looking at the GUI. This could make it about as close to the actual user experience as any automated tool has managed yet.

Solution?

As frustrating as it can be, GUI-level automated testing is still pretty new and still has some growing up to do. The tools are improving as GUI-level automation becomes more popular. It will be interesting to see what direction it takes.

11 thoughts on “The trouble with GUI automation

  1. Pingback: Tweets that mention The trouble with automation | Purple Box Testing -- Topsy.com

  2. Ron Pihlgren

    Nice post. I don’t work in the web UI world but what you say resonates. IMO, GUI automation isn’t something that can’t be done well but I don’t think it can be done with most of the GUI automation tools that we’ve had until now. I expect it will get better but not by people avoiding it. Also, as you mention, people tend to underestimate it; think that it should be simple but in reality it’s a difficult problem. Which means there are interesting times ahead in the space.

  3. Pingback: Blog Post: The Trouble with Automation | IT.beta

  4. Lisa Crispin

    You make some excellent points. I’ve seen so many teams create huge suites of GUI tests that are so poorly designed, they’re impossible to maintain.

    There ARE some good tools, but even with those, good code design practices are required. My team started using Canoo WebTest back in 2003 for GUI regression tests – they have very little logic, they mainly test happy path and obvious error conditions. We refactor them constantly to try to keep duplication to a minimum and keep each test to a clear purpose. Our ROI on these has been excellent – maintenance effort is low, and they do catch regression failures. Of course, we also have extensive automated regression tests at the unit level and behind the GUI.

    Tools are getting better and better all the time. We have some awesome Watir scripts to help us with manual exploratory testing. I’ve been impressed with Robot Framework, Cucumber, Selenium (and its many implementations such as Selenesse) and SWAT. But they all require good design skills. Programmers need to get involved with designing the automated tests, and teach the testers how to ensure they’re maintainable.

  5. Pingback: QA Hates You » Blog Archive » How Can You Tell An Experienced Automated Tester?

  6. Trish Khoo Post author

    Hi Lisa, I completely agree – good code design practices are required and this isn’t always acknowledged. Poor tools are only a small part of the problem, the biggest maintenance overhead is caused by poor design.

    The tools are definitely getting better, but they still have a long way to go.

  7. Trish Khoo Post author

    Thanks Ron. You’re right, it should be much simpler than it is. I think that is why people underestimate its difficulty, and I don’t think it is wrong to expect it to be easier.

  8. Ron Pihlgren

    >>I don’t think it is wrong to expect it to be easier.

    Not wrong / not right. But whatever the expectation, it isn’t easy (yet).

    Your post got me thinking. I may actually have to write a post on my (poorly-maintained) blog…

  9. Albert Gareev

    Just a little rant.

    “bloated overpriced commercial tools”
    – I suspect this myth is continuously cultivated for certain reasons by some people.
    Come on! Compare USD 2K (many really powerful tools are offered for this price, including TestComplete) with other software or hardware pricings, or test consultants’ rates.

  10. Jim

    GUI automation tools have been around since ’93 with Test32 by microsoft. SilkTest etc.. been around since ’98.

    It’s not new. The GUI’s under test have changed, the tools haven’t much.

    SilkTest
    WinRunner (now QTpro)
    Rational’s functional Tester (IBM may call it something diff now)
    Watir (newest really, and ruby based. Teething pains just like SilkTest went through in the beginning).
    Selenium (been around for a while now)
    etc…..

    GUI automation relies on sound up front design/implementation by developers so that the app can be GUI tested. Most web based companies use cowboy programmers, very little process and thereby incur massive technical debt around their production platform. Since you won’t really improve this unless the company has a serious roadmap for maturing, GUI automation has low ROI. You are writing software to test software. All rules of sound software development hold true.

Leave a Reply