The death of quality

“Quality is dead,” declared Goranka Bjedov. And I thought to myself, I knew it.

For some time now I’ve been concerned about this. I’d always seen specialised testing as a way to turn a mediocre-quality product into a high-quality product. But the evidence before my own eyes is that the market is full of buggy software. The worst part is, people accept buggy software as the norm and learn to live with it.

We say that nobody can ever be “done” testing, because there is always more than can be tested. There comes a point in any project where the cost of continuing to test outweighs the benefit in releasing earlier, but with unknown bugs.

“The truth is, bad software makes more money” – Goranka Bjedov, STANZ 2011.

I recently watched a project going through a fairly difficult stabilization phase. Too much change towards the end of the development cycle and not enough testing at the start meant that most of the bugs were being found at the end, so the team was looking at a few more weeks of catch up time to find and fix all of the bugs. But, for reasons that I still don’t completely understand, all of the bugs were postponed and it was released anyway. The next two weeks were a mad scramble to fix production bugs, followed by several weeks of less frantic fixing. I was ready to label the project a colossal failure. But the product was actually very well-received by most customers. By the end of the first week, customers were singing praises for the new feature and the project was hailed as a success.

It left me wondering what the point was of having testers on that project at all. I was told that, without testers, it would have been a whole lot worse. Is that rewarding though? Working to make something “not as bad as it could be”?

After STANZ, I stayed with my friend Richard for a few days. Richard is a medical doctor, training in radiation oncology. I asked him if it was depressing being in a job treating cancer, which is a condition that’s difficult to treat and not completely curable.

“Oh not at all” he said. “There are some practical things you can do, to improve quality of life”.
“But,” I pressed, “when people think cancer, they usually think that’s it for them. It seems like you’re just fighting an uphill battle.”
“Everyone dies eventually, of something,” he explained. “But I know that there’s things I can do that means they’re not going to die today. And the quality of life they have in the meantime can be made a lot better.”
I nodded. “I guess that’s true.”
“Once you accept mortality, medicine’s not that hard. It’s actually quite rewarding.”

Perhaps we need a mentality shift, and a different way of seeing our role. If we see ourselves as the cure for bad software, then we’re going to be repeatedly disappointed.

I emailed my friend James Martin for his perspective, and he offered this one:

How about working to make something ‘better’. What if all any of us did, regardless of job title, was try as hard as we could to make the best stuff possible, accepting the fact that even the best of us has only a fragile grasp of what it takes to make something that another person will be delighted by.

Personally, it helps me to think that my whole job description is ‘to make things better’. That way I can solve problems using whatever tools feel right for the task at hand and I don’t feel constrained by someone else’s opinion of who I should be and what I should do.

As usual, the people setting the ‘standards’ for our ‘profession’ do so without knowing our context. But we can set our own standards for ourselves and play by our own rules. I like to think that you and I did that last year. I still think of what we did at Campaign Monitor as my best work.

If quality is dead, what are we doing here?
If we are adding value, what is it and how do we measure it?

15 thoughts on “The death of quality

  1. Dewi

    “The truth is, bad software makes more money”

    Yeah, so can fraud. And this is a class of fraud. Unless a company is being absolutely up-front with all stakeholders and end users entrusting their data to it, there is no excuse for failing reasonable quality standards. For things like privacy and security, even attempting to disclaim responsibility wouldn’t even be legal in many jurisdictions.

    Incidentally, let’s run with the cancer medicine example: look at the figures for ALL, a childhood cancer. In the 60s the 5-year survival rate was 10%. Every 5 years since it has climbed significantly, to 96% in the early 2000s.* That’s what scientific & technological progress looks like.

    Of course most of us are not building software that lives depend on, but it’s safe to say software plays a pivotal role in everyone’s lives and the influence is growing fast. If chasing quality seems difficult, ubiquity & sophistication of new systems might have something to do with it. We’ll never get to defect-free, but it’s critical that process development continues, if only to keep up.

    * Source: http://scienceblogs.com/pharyngula/2011/03/support_cancer_research_now.php

  2. Andrew Prentice

    “The truth is, bad software makes more money”…provided the bad software is given away for free and the money is made out of exploting its users (and not the users exploiting the software), right?

    I don’t see any evidence that quality is dead. What I do see is the timeframe for quality becoming more elastic i.e. quality being obtained through many smaller and more frequent releases instead of very infrequent larger ones. That allows us to increase quality over time, via releases that can be (not not necessarily) of lower quality.

    Quality is an infinite work in progress. It always has been. I think we’re just adjusting to seeing it as such.

  3. Christian Baumann

    Hi,

    impressive article, and interesting thoughts, thanks.

    I agree to the “The truth is, bad software makes more money”-statement, and can somehow understand it, and what it means for us testers.

    But…
    I don´t think that the cancer-example is fitting completely.
    Doctors are doing the best they can, applying the best methods they know, to help people to cure cancer.
    But we (testers, programmers, project managers) have better tools, and more knowledge, methodologies, techniques, etc. to improve the quality of software products. Very often they´re “just” not applied (testing more early in the product lifecycle, doing TDD, etc. …).

    So imho there´s a significant difference between software testing and curing cancer, which makes the comparision only senseful to a certain extense.
    I totally understand the doctor´s point of view when he´s saying that he´s improving the quality of a person who has cancer, and that this sometimes is the only thing he can do.
    But software testers (together with everybody else developing the prodcut) could do more then just making a bad software a little bit less buggy. Imho we could improve the overall quality significantly, by just applying more what we know & have in place already.

    Best Regards,
    Christian

  4. Ben Kelly

    That word Quality – nebulous little bugger isn’t it?
    Jerry Weinberg says that quality is value to some person (and I believe it was Michael Bolton and Markus Gaertner respectively who added ‘that matters’ and ‘at some point in time’).

    So if that’s the case, why do so many people seem to equate ‘Quality’ with ‘Perfection’?
    It something is not ‘error free’ and shiny, it is not a quality product. Or so any number of IT professionals I’ve worked with would have you believe – at least until you start digging.

    Once you scratch the surface and start asking tricky questions, then this lovely idealist facade of perfect software falls away. Does that mean quality is dead? I would say rather that quality exists in a form other than what you are expecting it to be, probably because no single one of us is able to say what quality *is* because it is not the same for everyone.

    I call it the ‘why ugly people can manage to get laid’ heuristic.

  5. Phil Kirkham

    When was quality ‘alive’ ? Remember the days of Blue Screen of Death or Sad Mac icons ? DLL hell ? Why did Jerry Weinberg title his book “Perfect Software and other illusions about testing ” ?

  6. Jeff Lucas

    I think it was Hume who said that experience is a system of beliefs and not knowledge (or fact). If you define quality in terms of the user experience, then you have to be careful in equating “bugginess” to quality.

    If the customer believes the software to have quality, then isn’t that enough? If, as a tester, If I apply a set of rules (tests, expectations) and those do not match the user experience then my conclusions are false. Knowing which bugs are important is the real challenge.

    As an example, I composed this on Tomboy notes in Ubuntu on an old laptop. As a tester, that combo has a lot of bugs but as a user it provides the most lightweight, agile OS I have available and I wouldn’t trade it for the world!

  7. Pingback: Is quality really dead? - ^(testing yet)(!|?)$

  8. Trish Khoo Post author

    Thanks for all the good thoughts guys.

    I’ve seen this quoted a few times since posting: “Quality is value to someone that matters at some point in time.” I’m familiar with the quote and I agree with it.

    I guess “quality” isn’t really the issue to me here. I wonder, can we build a *good enough* product without testers? If so, where does that leave testers?

  9. Bruce McLeod

    When feature parity exists, the product with the higher quality for a given price will win. Just look at Apple’s bottom line and tell me that quality is not a major factor in that.

    But when product that never ships, or doesn’t have feature parity, then quality never has a chance to be the differentiator.

  10. Zhuang Xu

    Thanks for the great article, Trish. Very inspiring.

    “I wonder, can we build a *good enough* product without testers?” – I share the same thoughts of James Martin – “… I can solve problems using whatever tools feel right for the task at hand and I don’t feel constrained by someone else’s opinion of who I should be and what I should do.” – If my and other team members’ job titles can be changed from “tester” or “developer” to “someone who makes things better’, then by default I will be authorized and trusted to fix bugs that I found, and the developers can also be trusted to test their works, because we both have enough skills and knowledge to both test and develop. If the difference in our strength for these two types of work, if there is any, and also each person’s opinion in what quality means, are not big enough to compromise the “quality” of either type of works, then some overheads coming with aligning each person’s rigid and sometimes vague job descriptions with the final outcome that really matters to customers (which is good enough value for money imho) could be avoided, and we will end up with less anxious testers and developers, and happier customers. (BTW I am not really sure if that’s practical to exercise in reality though.)

  11. Thomas Ponnet

    Hi Trish,
    very good post and one that I had to think about a bit before posting.

    To address your follow up question first, “… can we build a *good enough* product without testers? If so, where does that leave testers?”

    What is a tester other than a label that we put on a particular person working on a set of tasks that are very roughly in a similar area? If you ask what a tester at Microsoft does you’ll probably get a difference answer to what a tester at your or my company does.

    My point is that I believe that a role in a project is necessary to dig up information about the product to be shipped (if it’s a shippable product, there are others). That information, amongst information from other roles is then used to make a decision if that product is shipped or not, i.e. is it “good enough”. If you want to call that person a tester, SDET, a or someone else doesn’t really matter. At the root of it is, what decision makers need is information.

    About the statement “Quality is value to someone that matters at some point in time.” I agree with it as well.

    Recently I have found that when talking to fellow testers that the value part is quite subjective and that many believe that business representatives don’t have the right measure of what that value consists of, which is usually something along the lines of “How much money did we make (in the short term).”

    A common argument seems to be that businesses should consider the long term impact of shipping too early and that usability may be impacted by subjectively buggy software. As you pointed out, the reality seems to be that many end users annoyingly don’t seem to notice. And if the end users like it, the business makes money it’s hard to argue that “It could be even better.”

    The scary part for me about Weinberg’s Quality sentence is that if no one can tell us exactly who everyone is that matters, the business users (as example of people who matter most in reality) can’t tell us what their values are, does this not mean that the ship/not ship decision is purely an emotional one?
    In a way that can be considered good news – that’s where I see professional, experienced testers coming in, helping the Business with that decision.

    I’m interested to continue that discussion, feel free to ping me on twitter, email or my blog.

  12. nafees

    Hi,

    impressive article, and interesting thoughts.

    I agree to “Christian” for experienced thoughts that, “The truth is, bad software makes more money”-statement, and can somehow understand it, and what it means for us testers.

    But…
    I don´t think that the cancer-example is fitting completely.
    Doctors are doing the best they can, applying the best methods they know, to help people to cure cancer.
    But we (testers, programmers, project managers) have better tools, and more knowledge, methodologies, techniques, etc. to improve the quality of software products. Very often they´re “just” not applied (testing more early in the product lifecycle, doing TDD, etc. …).

    Cancer may not be cured at All, But a software is possible to be taken to its Utmost Quality Level, using provided toold, believes & experience.

    Thanks & regards.

  13. Johan Martinsson

    A thought of mine : people complain about bugs when the software doesn’t solve their problem. Sometimes this is due to too many bugs that have to be worked around. Sometimes the software are not fulfilling their needs.

    So my take would be that in the first case, quality (in the less bugs sense) is important but in no way a conditional for success. In the second, less bugs will hardly make the customers complain less.

  14. Johan Martinsson

    “Where does that leave testers”

    Given that lack of bugs is not a primary differentiator, I think the value of testers lies *a lot* in the definition of what to develop.

  15. Lisa Crispin

    I’m finding this a bit late thanks to @yorkyabroad’s links.

    I’ve heard people quote Jerry Weinberg as saying something like, just try to improve something 10% or 20% or something like that. OK, I can’t quote it well but the point is, improving something a little is worthwhile.

Leave a Reply