Off the beaten track

You learn a lot more when things don’t quite go to plan, than when everything goes smoothly. So here’s a story I think we can all learn from.

James and I wanted to do a kind of performance test which involved the creation of many elements in our system. So we decided that a quick way to do it would be to leverage the common methods in our existing GUI automation suite to create a heap of accounts through the interface. We drew up a plan and wrote the test script.

Why did we choose to use GUI automation? First, as was mentioned, we already had existing code we could reuse, so we thought script creation would be reasonably fast. Also, James hadn’t used our test suite much before, so it was a good opportunity for him to become more familiar with it. In addition we figured it might reveal some interesting GUI bugs while it was running, as kind of a by-product.

This test script was massive. Luckily, most of the creation methods were already written, so we just had to get the test creating what we wanted. I knew that GUI automation could be a bit fragile. So we got the script to track its progress in a text file, so that when it got interrupted, we could just restart it and it could just pick up from where it left off.

We estimated it would take a couple of days to create the script, and about 50 hours to run it. How wrong we were.

The first day started out slow, as James battled with installing Visual Studio, Webaii and Gallio on his machine, which progressed as an epic series of fails until finally he was able to START writing code at the end of the day. Meanwhile, I managed to make a decent start on my part of the script. After a while we realised we would have to write some of the data directly to the database, so we asked a friendly developer for some help. Writing data this way would prove to be a bit of a time saver later.

Four days later, we were finally able to start running this script. It tripped over a few times, but it could run uninterrupted for an hour or two. Some observations were made:

TK: a little null reference exception never hurt anyone. off it goes again.

JM: As if nothing ever happened

TK: happy, carefree test script

TK: gamboling through the wild plains of GUI automation

TK: though null reference exceptions may block its path, does it falter? nay it continues, unhindered and free

a majestic beast

It was actually running quite smoothly on my machine, only occasionally failing to find page elements for some unknown reason. So we ran it on my machine for about half a day.

JM: Sweeping majestically across the plains and all that…

TK: like a gazelle, leaping through the DOM…

JM: Yeah!

JM: Definitely evoking gazelle-like feelings

TK: let no id=basicTab stand in its way

TK: it’s much nicer to think of it as a gazelle

JM: Than?

TK: rather than how i usually think of it – like a small child picking up the browser and punching it into submission

After a few hours we realised that our original estimates were way off. At this rate it was going to take about 250 hours to run. We didn’t have that kind of time, so we decided to split up the creation workload and run it in parallel across five different machines. Then the real fun began. While it ran relatively smoothly on my machine, it didn’t go so well on other machines because my machine was much faster than the other machines we acquired. So pages were taking longer to load on the other machines, and the automation kept falling over because it couldn’t find page elements in time. Our majestic script was not looking so gazelle-like anymore.

TK: off it stumbles, like a fumbling beast, tripping over its own legs and the occasional blade of grass

TK: running like an oafish monstrosity, crashing into trees and occasionally falling into holes that aren’t there

TK: oh see how on broken wings it flies, soaring erratically through slight zephyrs, smacking into small buildings

It would only run a few hours at a time if we were lucky, and it had to be manually restarted. After a few days, its progress was dismally short of its target.

We decided that we couldn’t spend much more time on this, so if it wasn’t done by the end of the day, we would just use what we had. We still wanted to get some nice peaks happening, so we modified the progress files to focus only on the peak areas, divided it up among the working machines and set it on its way again.

With most of the test data in place, we made plans to start our performance test. The final piece of the puzzle was simulating user activity in order to trigger the functionality that would (we hoped) stress the system in interesting ways. We took the path of least resistance in this case; using a Ruby library that had been written as an experiment in a previous test. With a few small modifications, the Ruby script consumed the output from our main setup automation and simulated the actions that we required. Again, we divided the work and ran several scripts in parallel to speed things up.

After running the test for a few days, we didn’t find any big showstopper bugs but we did end up finding a few smaller bugs. So was it worth doing? We didn’t necessarily get what we set out to achieve, but it produced many valuable outcomes:

• It demonstrated a few things about our test framework that could be improved upon, which motivated us to make these improvements.
• It demonstrated a few shortcomings of GUI-level automation. Trying to get GUI-level automation to do anything for a long amount of time is prone to error from multiple sources: The test environment, the browser, the speed of the machine it’s running on, the sun, the moon, the way MbUnit happens to be feeling at the time, etc. For generating data, we would have been better off just directly injecting the data into the database.
• Just by telling the development team that we were going to do this test, and telling them exactly what we were planning to do made them think about what issues it could run into and they started working on improvements even before the test had been run. One developer who was assigned to look into a specific bottleneck ended up using elements of our data setup script to create his own targeted test for his changes.
• The Ruby user action simulator ended up being a really useful general purpose tool.
• We discovered a heap of GUI-related bugs as a result of checking the progress of the GUI automation.

What could we have done better?

• When we started out, we only had a vague idea of how long it would take and how long we could afford to spend doing the test (we needed to ship the software at some point too). In hindsight, perhaps we should have set some clear upper limit on the time we would spend working on the problems before admitting defeat and attempting to tackle the problem in another way.
• Our objectives and expectations evolved as we learnt about the system, the test framework and hundreds of other variables that we hadn’t considered. This is a common problem in software projects and perhaps if we had set some clearer goals at the start we wouldn’t have drifted at times.
• We could have involved other team members more in what we were doing by updating them on the specifics of what we were actually implementing in order to give them a chance to give their feedback and opinions.

And hey, in the end the feature was deployed successfully and within a reasonable timeframe. So ultimately we achieved our mission, and learned some valuable things in the process. Success!

Written in collaboration with James Martin.