We work with a number of clients to help them implement risk based testing. It's important to keep the process simple enough for broad-based participation by all stakeholders. A major part of doing so is simplifying the process of assessing the level of risk associated with each risk item.
To do so, we recommend that stakeholders assess two factors for each risk item:
Likelihood arises primarily from technical considerations, such as the programming languages used, the bandwidth of connections, and so forth.
Impact arises from business considerations, such as the financial loss the business will suffer, the number of users or customers affected, and so forth.
We have found that project stakeholders can use these two simple factors to reach consensus on the level of risk for each risk item. These two factors are also sufficient to achieve the benefits of risk based testing that I discussed in an earlier post. Using more than these two factors tends to make this process overly complicated and often results in the failure of attempts by project teams to implement risk based testing.
It's important for project teams to select the right mix of test strategies for their projects. In our training, outsourcing and consulting work, we typically see one or more of the following strategies applied:
It's important to remember that strategies may be combined; test managers should employ all of the strategies that they can effectively and efficiently employ on a given project. Test managers should carefully select and tailor the strategies they use for a project.
Since software testing is an assessment of quality, this question is not a theoretical one. I suggest using a definition from J. M. Juran, one of the quality gurus who helped Japan achieve its astounding progress in the last 60 years. Juran wrote that "quality is fitness for use. Features [that] are decisive as to product performance and as to 'product satisfaction'... The word 'quality' also refers to freedom from deficiencies…[that] result in complaints, claims, returns, rework and other damage. Those collectively are forms of 'product dissatisfaction.'" Since satisfaction revolves around the question of who is (or isn't) satisfied, it's clear that Juran is referring to the satisfaction of key stakeholders.
Okay, that's pretty straightforward. So, how do we think about quality? I suggest there are three ways to approach this question:
So, if we think about quality properly, and think about the approach to thinking about quality properly, then we can approach the assessment of quality properly.
Here are four simple rules for good software testing metrics:
While simple to state, these rules are important. My associates and I find that, almost every time there is a test results reporting problem in an organization, one or more of these rules is being violated. Follow these rules for better software test metrics, and thus more effective software test management.
One of the great frustrations for testers, test managers, and other project team managers is the elusive test execution completion date. It always seems to take longer than you'd think to get done with test execution, especially on larger projects.
So, how do you predict when will you be done executing the tests? Part of the answer is when you’ll have run all the planned tests once. This involves knowing three things:
This figure sets a minimum time required to finish test execution.
However, test execution can last longer than this, because the other part of the answer is when you’ll have found the important bugs and confirmed those bugs to be fixed. This involves using historical data (or extremely good guesses) to determine four things:
Obviously, solid historical data from similar past projects really helps with this kind of estimation. For our clients who have such data, and formal defect removal models, they can get quite accurate, often predicting the end date of test execution to within plus or minus 10% even on test execution efforts that last over six months.
As some of you might know, I'm a big proponent of risk based testing. (For example, see the podcasts and videos on the RBCS Digital Library here.) In fact, a major RBCS client--I can't mention their name but the odds are good that you own one or more of their products--just told us that an entire division of their enormous company is adopting risk based testing, based on their understanding of the technique from our Advanced Test Manager course.
When test professionals first learn about risk based testing, one important question that often comes up is, "How do I convince skeptical testing stakeholders (outside of the test team) that risk based testing of our software is smart?" You can give them a whole lecture in response to this question, but long answers tend to produce a severe case of MEGO ("my eyes glazed over") in non-test people.
In business, people talk about the "elevator pitch." If you haven't heard this phrase, here's what it means: You have a powerful executive in an elevator with you. She's getting off in about 10 floors. You have just a few seconds to convey to this powerful person some important piece of information. Start talking.
So, if you find yourself in an elevator, a conference room, or an office with an influential testing stakeholder, and you want to convince them to support your efforts to implement risk based testing, here's the elevator pitch:
All of these benefits allow the test team to operate more efficiently and in a targeted fashion, especially in time-constrained and/or resource-constrained situations.
Are you a test manager estimating a new software or systems test project? Here's a list of ten factors that will make the testing cost more, take longer, or both.
Don't forget about these factors when you do your estimate. Just one of these factors can cause some real pain and suffering when the consequences arise.
After years of hesitation, RBCS is finally initiating a blog on software testing. We've hesitated in large part because I've always joked--and only half in jest--that the word "blog" seems to rhyme with "blab" given most blog content.
However, as the cliche goes, better to light a candle than to curse the darkness. In this blog, I'm going to focus on sharing immediately useful ideas about software testing, rather than merely opinions, bloviations, or jeremiads. I'm also going to talk about what you tell me you want to hear about, so feel free to send me requests.
In the next couple days, I'll start with a series of blog postings on some key management concepts related to software testing. I'll be happy to see comments on those, and we can use the blog postings as the start of a discussion of each of these concepts.
President, RBCS, Inc.