I find that most test methodologies today don’t fit very well with agile development methodologies. Whenever requirements change, the developers adapt fairly easily but the testers are still gritting their teeth because they have to update a huge backlog of test cases for regression testing.
Most test orgs do try to do similar tasks at the same time, like write test cases while developers are doing design. I think the problem is that many people think test cases are equivalent to design documents, but they’re not – they’re more equivalent to pseudocode or Z-specification, which says exactly what you’re going to do before you do it. Imagine if you wrote pseudocode for all your apps before you wrote actual code, then had to go back and update all the pseudocode after anything changed. You’d think “what a waste of time, why aren’t I updating the actual code?”
The test APPROACH is more equivalent to the design documents – it outlines how you will test, not the items you will click in order to do it.
Why remove test cases?:
From Lessons Learned in Software Testing
Lesson 46: One important outcome of a test process is a better, smarter tester.
We often hear arguments against any form of testing that results in minimal or no documentation, as though the only value of testing is what comes from writing down our tests. This ignores a profoundly important product of testing: the tester herself.
Good testers are always learning. As the project progresses, they gain insight into the product and gradually improve their reflexes and sensibilities in every way that matters in that project. An experienced tester who knows the product and has been through a release cycle or two is able to test with vastly improved effectiveness – even with out any instructions – compared to an inexperienced tester who is handed a set of written instructions about how to test the product.
Some consultants and writers in the fields seem to believe that an ineffective tester can be transformed into an effective tester merely by handing her a test procedure. In our opinion, this is a bad practice. It reflects a fundamental misunderstanding of testing and the people who do it well.
Goals of XT:
– Turn the main defect-finding activities up to maximum, cut out all the non-essential bits.
– Keep up with agile processes without a ton of maintenance required.
– Encourage tester thinking and creativity during the actual testing process.
– Highlight important information, when it matters.
– Increase test coverage.
– Eliminate risk related to out of date test cases.
– Encourage greater understanding of product requirements.
– Reduce focus on CRUD functionality (basic functionality which is expected to work), more emphasis on testing integrated system components and complex scenarios
Concepts of XT:
– Less time spent documenting means more time for testing. More time for testing means more defects found.
– Without defined test script steps, testers can use their own skills and judgement to decide the best way to test the requirement when the product is presented to them.
Process of XT:
1. Read requirements, familiarise yourself with them. Break them down into smaller chunks and add assumptions and implied requirements if necessary.
2. Conduct a risk analysis to identify which requirements are highest priority.
3. Receive application.
4. Test against requirements. Log pass/fail results against them. Relate defects to them.
5. Create a regression suite from high priority requirements and high priority defects.
Metrics from XT:
– Map defect clusters to requirement areas
– % pass/fail requirements
– number of defects
– Testing directly against latest requirements eliminates time spent updating test cases. Eliminates risk of testing against out of date test cases.
– Saves time writing test cases, the majority of which will be discarded later anyway.
– Makes requirement coverage easier to track.
– More exploratory testing is encouraged.
– More tester creativity is utilised at a time when the most information is available to the tester.
– Reduces the false sense of security created by a sea of low priority test cases that pass
– More agile – can keep up more easily with changed requirements
– Cannot get non-testers to run scripts, as the methodology requires a medium to high level of tester experience.
– Hard to track specifics of passed tests, as details are only recorded for failed tests (in defects)
– Hard to produce proof of testing when most of the tests pass.
– Managers and customers may find the results alarming and overly negative, which may lead to undesirable consequences.
Thoughts? Flames? Rapturous cries of agreement?