Resources
Our notable partnerships give us access to the best resources and tools for the job.
Get software testing training resources:
Risk-based testing is an approach to testing that optimizes the reduction of risk to the quality of the product, allocating test effort and prioritizing test execution based on risk. It not only reduces the residual level of quality risk but also informs stakeholders of the residual level of risk, starting in the initial stages of a project. It involves the identification of product quality risks and the use of risk levels to guide the test process.
Resources for “Risk Based”
Articles
More and more projects involve more integration of custom developed or commercial-off-the-shelf (COTS) components, rather than in-house development or enhancement of software. In effect, these two approaches constitute direct or indirect outsourcing of some or all of the development work for a system, respectively. While some project managers see such outsourcing of development as reducing the overall risk, each integrated component can bring with it significantly increased risks to system quality. Read this software testing article to learn about the factors that lead to these risks ,and strategies you can use to manage them.
Since it is not possible to test everything, it is necessary to pick a subset of the overall set of tests to be run. Read this article to discover how quality risks analysis can help one focus the test effort.
This article presents a case study of a risk-based testing pilot project at CA, the world's leading independent IT management software company. The development team chosen for this pilot is responsible for a widely-used mainframe software product called CA SYSVIEWR Performance Management, an intuitive tool for proactive management and real-time monitoring of z/OS environments. By analyzing a vast array of performance metrics, CA SYSVIEW can help organizations identify and resolve problems quickly.
CA piloted risk-based testing as part of our larger effort to ensure the quality of the solutions we deliver. The pilot consisted of six main activities:
- Training key stakeholders on risk-based testing
- Holding a quality risk analysis session
- Analyzing and refining the quality risk analysis
- Aligning the testing with the quality risks
- Guiding the testing based on risks
- Assessing benefits and lessons
This article addresses each of these areas - as well as some of the broader issues associated with risk-based testing. Click here to read the version of this software testing article as published in Better Software Testing.
Rex Black’s pioneering Managing the Testing Process was both the first test management book and the first to discuss risk-based testing. In this software testing article, Rex explains:
- The benefits of risk-based testing.
- Why adding risk analysis to the test team’s responsibilities actually reduces their workload.
- The importance of stakeholder participation.
- Common mistakes that can occur in risk-based testing.
Rex illustrates these points, not through hypothetical discussion, but by examining a case study where RBCS helped a client launch risk-based testing. Read this article to learn how to analyze risks to quality, and use that analysis to be a smarter test professional.
Analytical risk based testing offers a number of benefits to test teams and organizations that use this strategy. One of those benefits is the opportunity to make risk-aware release decisions. However, this benefit requires risk based test results reporting, which many organizations have found particularly challenging. This article describes the basics of risk based testing results reporting, then shows how Rex Black (of RBCS) and Nagata Atsushi (of Sony) developed and implemented new and ground-breaking ways to report test results based on risk.
Testing can be thought of as (one) way to reduce the risks to system quality prior to release. Quality risks typically include possible situations like slow system response to use input, incorrect calculations, corruption of customer data, and difficulty in understanding system interfaces. All testing strategies, competently executed, will reduce quality risks. However, analytical risk based testing, a strategy that allocates testing effort and sequences test execution based on risk, minimizes the level of residual quality risk for any given amount of testing effort.
There are various techniques for risk based testing, including highly formal techniques like Failure Mode and Effect Analysis (FMEA). Most organizations find this technique too difficult to implement, so RBCS typically recommends and helps clients to implement a technique called Pragmatic Risk Analysis and Management (PRAM). You can find a case study of PRAM implementation at another large company, CA here. While this article describes the implementation of the technique for projects following a sequential lifecycle, a similar approach has been implemented by organizations using Agile and iterative lifecycle models.
Analytical risk based testing offers a number of benefits to test teams and organizations that use this strategy. One of those benefits is the opportunity to make risk-aware release decisions. However, this benefit requires risk based test results reporting, which many organizations have found particularly challenging. This article describes the basics of risk based testing results reporting, then shows how Rex Black (of RBCS) and Nagata Atsushi (of Sony) developed and implemented new and ground-breaking ways to report test results based on risk.
Testing can be thought of as (one) way to reduce the risks to system quality prior to release. Quality risks typically include possible situations like slow system response to use input, incorrect calculations, corruption of customer data, and difficulty in understanding system interfaces. All testing strategies, competently executed, will reduce quality risks. However, analytical risk based testing, a strategy that allocates testing effort and sequences test execution based on risk, minimizes the level of residual quality risk for any given amount of testing effort.
There are various techniques for risk based testing, including highly formal techniques like Failure Mode and Effect Analysis (FMEA). Most organizations find this technique too difficult to implement, so RBCS typically recommends and helps clients to implement a technique called Pragmatic Risk Analysis and Management (PRAM). You can find a case study of PRAM implementation at another large company, CA, here. While this article describes the implementation of the technique for projects following a sequential lifecycle, a similar approach has been implemented by organizations using Agile and iterative lifecycle models.
This article was originally published in Software Test and Quality Assurance www.softwaretestpro.com in their December 2010 edition.
In the Pragmatic Risk Analysis and Management process described in books such as Managing the Testing Process, Pragmatic Software Testing, and Advanced Software Testing: Volume 2, I define the following extents of testing, in decreasing order of thoroughness:
- Extensive
- Broad
- Cursory
- Opportunity
- Report bugs only
- None
Risk based testing does not prescribe specific test design techniques to mitigate quality risks based on the level of risk, as the selection of test design technique for a given risk item is subject to many factors. These factors include the suspected defects (what Boris Beizer called the “bug hypothesis”), the technology of the system under test, and so forth. However, risk based testing does give guidance in terms of the level of test design, implementation, and execution effort to expend, and that does influence the selection of test design techniques. This sidebar provides heuristic guides to help test managers and engineers select appropriate test techniques based on the extent of testing indicated for a risk item by the quality risk analysis process. These guides apply to testing during system and system integration testing by independent test teams.
More and more projects involve more integration of custom developed or commercial-off-the-shelf (COTS) components, rather than in-house development or enhancement of software. In effect, these two approaches constitute direct or indirect outsourcing of some or all of the development work for a system, respectively.
While some project managers see such outsourcing of development as reducing the overall risk, each integrated component can bring with it significantly increased risks to system quality. In this article, I’ll explain the factors that lead to these risks, and then strategies you can use to manage them.
I’ll illustrate the factors and the strategies with a hypothetical project. In this project, assume you are the project manager for a bank that is creating a Web application that allows homeowners to apply for a home equity loan on the Internet. You have two component vendors. You buy a COTS database management system from one vendor. You will hire an outsourced custom development organization to develop the Web pages, the business logic on the servers, and the database schemas and commands to manage the data. Let’s see how you can recognize the factors that create quality risks, and the strategies you can use to manage those risks.
Testing any real-world system is potentially an infinite task. Of this infinite set of possible tests, test managers need to focus on the most significant risks to system quality. These are the potential failures that are likely to occur in real-world use or would cost a lot if they did occur. This article describes practical ways to analyze the risks to system quality, providing guidance along the way to achieving effective and efficient testing.
Risk is a highly studied subject:
- 249 books found on an Amazingly Large Bookstore’s Website
- Hundreds of papers, conference proceedings, tutorials, classes, and book chapters
But, few of us understand the human side of risk
If you are a software developer, software development manager, or software quality assurance staff member, you probably know that developing secure software is no longer simply desirable—it’s completely essential.
Recently, we worked on a high-risk, high-visibility system where performance testing (“Let’s just make sure it handles the load”) was the last item on the agenda. As luck would have it, the system didn’t handle the load, and very long days and nights ensued. Delivery was late, several serious disasters were narrowly averted, and large costs were incurred.
Before we can build a high-fidelity test system, we have to understand what quality means to our customers. Test professionals can avail themselves of three powerful techniques for analyzing risks to system quality. Targeting our testing investment by increasing effort for those areas most at risk results in the highest return on investment.
If you are a software tester, programmer, or manager, you probably know that developing secure software is no longer simply desirable—it’s completely essential.
Some might assume that most security problems arise from the operating system or networking layers, well below the application code they are working on. However, figures for Web-based applications show that over three-quarters of security exploits arose from applications.
So, you know you need secure code, but how to get there? What are your security risks? What security failures and bugs do you have? What do these security risks, failures, and bugs mean? How can you reduce security risk in a way that doesn’t create new problems? How do you monitor my progress over time? This article will outline seven steps that will allow you to answer these and other questions as you improve your software’s security.
This is an excerpt from my book, Expert Test Manager, written with James Rommens and Leo van der Aalst. I hope it helps you think more clearly about the test strategies you use.
A test policy contains the mission and objectives of testing along with metrics and goals associated with the effectiveness, efficiency, and satisfaction with which we achieve those objectives. In short, the policy defines why we test. While it might also include some high-level description of the fundamental test process, in general the test policy does not talk about how we test.
The document that describes how we test is the test strategy. In the test strategy, the test group explains how the test policy will be implemented. This document should be a general description that spans multiple projects. While the test strategy can describe how testing is done for all projects, organizations might choose to have separate documents for various types of projects. For example, an organization might have a sequential lifecycle test strategy, an Agile test strategy, and a maintenance test strategy.
[How can dumb monkeys built from free tools help you? Give this article a read to see a case study. Originally published in Software Testing Professional magazine in 2008, these ideas and techniques are still relevant to SDETs, Technical Test Engineers, and Technical Test Analysts looking to build their own automation solutions using open-source components.]
Arrowhead Electronic Healthcare has been creating eDiarys on handheld devices since 1999. Arrowhead helps pharmaceutical research and marketing organizations document important information about how their products are being used in patients’ homes.
ePRO-LOG is Arrowhead’s third generation eDiary product. The primary design goal of ePRO-LOG is to be able to rapidly deploy diaries used for data collection in clinical trails and disease management programs.
A typical diary may include 100 forms translated in 15 or more languages, and used in several locales. This results in a large number of software builds and configurations. As a result, we needed an automated test tool to address potential risks and to automate common tasks.
The most important quality risks we wanted to address were:
- Reliability
- Translation completeness
- Functionality of UI
- Input error checking
- Verification of requirements
We needed an automated test tool with the following capabilities and features:
- Address defined risks
- Produce accurate form-flow diagrams
- Reduce tedium and opportunity for error in manual testing
- Save effort associated with manual testing for these risks
- Improve time-to-market by reducing test cycle duration through 24x7 testing
- Provide auditable documentation
- Handle any screen flow or translation without custom test scripts (i.e., be trial-independent)
- Be easy to implement and cost effective
This is a case study in how we reduced our risks and achieved our test automation objectives in just a few months on a total tools outlay of $0.
Webinars
Risk Based Testing Videos: How do You Know You Did Risk Based Testing Right (IV)
Recorded November 18, 2010
Risk Based Testing Videos: How do You Know You Did Risk Based Testing Right (III)
Recorded November 3, 2010
Risk Based Testing Videos: How do You Know You Did Risk Based Testing Right (II)
Recorded October 24, 2010
Risk Based Testing Videos: How do You Know You Did Risk Based Testing Right (l)
Recorded October 7, 2010
Risk Based Testing Videos: Risk Based Effort Allocation Test Sequencing and Test Selection
Recorded January 2, 2010
Podcast Episodes
Webinar: Two Bug Metrics, Millions in Process Improvement
Length: 1h 20m 50s
When we do assessments, we always try to look at process metrics. In most cases, we can find millions of dollars in process improvement opportunities. In this webinar, Rex will show you how two very simple bug metrics, calculated using only two simple facts for each bug report using simple, free spreadsheets you can get from our website, can reveal millions and millions of dollars in potential process improvements. All the more reason to track those bugs! To paraphrase Timothy Leary: Tune in, download, and drop software co
The Most Dangerous Fallacies of Risk-based Testing
Length: 0h 43m 17s
Mark Twain once wrote, “It ain’t what you don’t know that gets you into trouble. It’s what you know for sure that just ain’t so.” In logic, that thing you know that just ain’t so is called a fallacy. Fallacies abound in software testing, and risk-based testing especially suffers from a number of misconceptions. Are you laboring under such delusions? If so, what damage is that doing to your ability to implement the most effective and efficient way to manage the risk to the quality of your software and systems? In this free webinar, Rex will identify and dispel these fallacies, and take your questions about risk-based testing fallacies that are affecting you.
Agile Risk-based Testing: A How-to Guide
Length: 1h 35m 52s
Perhaps you, like many software testers and test managers, are enjoying the benefits of incorporating a lightweight analytical risk-based testing strategy such as RBCS’ Pragmatic Risk Analysis and Management (PRAM) technique into your test approach. What if your organization is adopting an agile methodology? Can you still use PRAM and other similar analytical risk-based testing strategies? Absolutely. In this webinar, Rex will explain the process for integrating quality risk identification and assessment into the release and iteration planning processes, and for integrating quality risk mitigation into each iteration. You’ll return to work with a proven method for injecting risk-based testing into your agile processes.
Training
ISTQB Virtual Advanced Security Tester Boot Camp
The Advanced Security Tester Boot Camp course, created by Rex Black, past President of the International Software Testing Qualifications Board (ISTQB), past President of the American Software Testing Qualifications Board (ASTQB) and co-author of a number of International Software Testing Qualifications Board syllabi, is ideal for testers and test teams preparing for certification in a short timeframe with time and money constraints.
All Rights Reserved.
All Rights Reserved.
PMI is a registered mark of the Project Management Institute, Inc.