Resources

Our notable partnerships give us access to the best resources and tools for the job.

Get software testing training resources:

Best practices are superior methods or innovative practices that contribute to the improved performance of an organization when applied with appropriate tailoring and under appropriate circumstances.  Best practices represent the current state of the art in a given industry.


Resources for “Best Practices”

Articles

When I wrote my book Critical Testing Processes in the early 2000s, I started with the premise that some test processes are critical, some are not. I designed this lightweight framework for test process improvement in order to focus the test team and test manager on a few test areas that they simply must do properly. This contrasts with the more expansive and complex models inherent in TPI and TMM.  In addition, the Critical Testing Processes (CTP) framework eschews the prescriptive elements of TMM and TPI since it does not impose an arbitrary, staged maturity model.

What’s the problem with prescriptive models?  In my consulting work, I have found that businesses want to make improvements based on the business value of the improvement and the organizational pain that improvement will alleviate. A simplistic maturity rating might lead a business to make improvements in parts of the overall software process or test process that are actually less problematic or less important than other parts of the process simply because the model listed them in order.

CTP is a non-prescriptive process model. It describes the important software processes and what should happen in them, but it doesn’t put them in any order of improvement. This makes CTP a very flexible model. It allows you to identify and deal with specific challenges to your test processes. It identifies various attributes of good processes, both quantitative and qualitative. It allows you to use business value and organizational pain to select the order and importance of improvements. It is also adaptable to all software development lifecycle models.

Continue reading →

Why Care About Testing Processes?

  • Bugs are expensive
    US economy loses $60B yearly due to bugs, $20B of which could be saved through better testing
  • Bugs drive away customers
    Handwriting recognition bugs in Newton cost Apple the person digital assistant (PDA) market
  • Bugs sometimes kill people
    Therac-25 overdoses, Patriot clock overflow
  • Inadequate testing endangers the project
    Bad testing processes can increase the risk of project delay or cancellation by 25 to 300% 

Continue reading →

As a test professional with almost two decades in the computer field, I’m pleased to see a new publication devoted to our field, the Journal of Software Testing Professionals, and honored to serve as a Contributing Editor. As my first task in this role, I have been asked to write and collaborate with other software testing professionals to produce a series of articles addressing what we consider critical processes that the test professional must master in order for her to succeed. In each of these articles, we’ll examine one critical process closely, offering practical advice on what works and caveats on what can cause problems

Continue reading →

So, you are responsible for managing a computer hardware or software test project? Congratulations! Maybe you’ve just moved up from test engineering or moved over from another part of the development team, or maybe you’ve been doing test projects for a while. Whether you are a test manager, a development manager, a technical or project leader, or an individual contributor with some level of responsibility for your company’s test and quality assurance program, you’re probably looking for some ideas on how to manage the unique beast that is a test project.

Continue reading →

In the last issue, I introduced this series of articles on critical software testing processes. For our first article, let’s look at the processes associated with an essential precondition for any test activities: creation, delivery, and installation of a software release into the test environment. Without software to test, there can be no software testing, but, as obvious as this is, many software development organizations do a poor job of managing test release processes. Often, the role of building a test release devolves into the test organization, which is generally ill-prepared to handle the tasks, and the processes are defined reactively— to put out fires—not proactively—to handle key roles optimally. 

Continue reading →

In the last issue, I dealt with a collaborative test process, getting test releases into the test lab. In this article, let’s look instead at an internal test process, managing the execution of tests against a test release. One can think of the test manager’s role during the test execution process as one of managing a search for the unexpected. While all managers are expected to manage unexpected events within their areas of responsibility, the test manager is the only manager in the software development organization whose area of responsibility is searching for the unexpected. This search takes the form of running a set of tests. How does a test team take these tests, run them against an installed system, and capture information about the system under test, the test cases, and the test execution process itself? 

Continue reading →

Pervasive testing means getting the right people working together through the right processes at the right time for high-ROI testing. Pervasive testing is the final piece of the test investment puzzle, and just as important as all the others. Through pervasive testing, all the ideas we’ve explored so far come together.

Continue reading →

Project Retrospective

By Rex Black

An excerpt from The Expert Test Manager: Guide to the ISTQB Expert Level Certification book by Rex Black, Jim Rommens and Leo Van Der Aalst due to be published by Rocky Nook. All material is provisional and may be subject to change
 
As a colleague told me once, a good motto for software teams is: "Make interesting new mistakes." His explanation was, since you are human, you'll make mistakes. But you should make interesting ones, ones you can learn from, and you should only make a mistake once.  How do you ensure that you make only interesting new mistakes? By learning from each mistake that you make. How to you learn from each mistake? In this short article, you'll read about a proven technique for learning from mistakes, retrospectives. Useful in both Agile and traditional lifecycles, this simple technique can make you and your colleagues a process improvement machine!
 

Continue reading →

"Back in 2010, at the launch of Core Magazine, http://www.coremag.eu/, I wrote a series of columns to welcome people to the magazine. As a sort of Throw-Back-December, here they are, as they appeared in the original magazine issues. I hope you enjoy them."
-Rex Black
 
Greetings, and welcome to my quarterly column on software testing best practices.  When I was asked to write this column, I had to choose the approach, the theme.  The writers' aphorism says, "Write what you know." So, what do I know?
 
Well, if you know me and my consulting company, RBCS, you know that we spend time with clients around the world, in every possible industry, helping people improve their testing with training or consulting services, or doing testing for them with our outsourcing services.  Our work gives me insights into what goes on, the actual day-to-day practice of software testing.
 

Continue reading →

Seven Steps to Reducing Software Security Risks

By Rex Black, President, RBCS, Inc.

If you are a software tester, programmer, or manager, you probably know that developing secure software is no longer simply desirable—it’s completely essential.  

Some might assume that most security problems arise from the operating system or networking layers, well below the application code they are working on.  However, figures for Web-based applications show that over three-quarters of security exploits arose from applications. 

So, you know you need secure code, but how to get there? What are your security risks?  What security failures and bugs do you have?  What do these security risks, failures, and bugs mean?  How can you reduce security risk in a way that doesn’t create new problems?  How do you monitor my progress over time?  This article will outline seven steps that will allow you to answer these and other questions as you improve your software’s security.

Continue reading →

This is an excerpt from my book, Expert Test Manager, written with James Rommens and Leo van der Aalst. I hope it helps you think more clearly about the test strategies you use.

A test policy contains the mission and objectives of testing along with metrics and goals associated with the effectiveness, efficiency, and satisfaction with which we achieve those objectives. In short, the policy defines why we test. While it might also include some high-level description of the fundamental test process, in general the test policy does not talk about how we test.

The document that describes how we test is the test strategy. In the test strategy, the test group explains how the test policy will be implemented. This document should be a general description that spans multiple projects. While the test strategy can describe how testing is done for all projects, organizations might choose to have separate documents for various types of projects. For example, an organization might have a sequential lifecycle test strategy, an Agile test strategy, and a maintenance test strategy.

Continue reading →

Test-Driven Development, Acceptance Test-Driven Development, and Behaviour-Driven Development

By by Rex Black, Marie Walsh, Gerry Coleman, Bertrand Cornanguer, Istvan Forgacs, Kari Kakkonen, and Jan Sabak

[Note: This is an excerpt from Agile Testing Foundations: An ISTQB Foundation Level Agile Tester Guide, by Rex Black, Marie Walsh, Gerry Coleman, Bertrand Cornanguer, Istvan Forgacs, Kari Kakkonen, and Jan Sabak, published July 2017. Kari Kakkonen wrote this selection. The authors are all members of the ISTQB Working Group that wrote the ISTQB Agile Tester Foundation syllabus.]

The traditional way of developing code is to write the code first, and then test it. Some of the major challenges of this approach are that testing is generally conducted late in the process and it is difficult to achieve adequate test coverage. Test-first practices can help solve these challenges. In this environment, tests are designed first, in a collaboration between business stakeholders, testers, and developers.  Their knowledge of what will be tested helps developers write code that fulfils the tests. A test-first approach allows the team to focus on and clarify the expressed needs through a discussion of how to test the resulting code. Developers can use these tests to guide their development.  Developers, testers, and business stakeholders can use these tests to verify the code once it is developed.

A number of test-first practices have been created for Agile projects, as mentioned in section 2.1 of this book. They tend to be called X Driven Development, where X stands for the driving force for the development. In Test Driven Development (TDD), the driving force is testing. In Acceptance Test-Driven Development (ATDD), it is the acceptance tests that will verify the implemented user story. In Behaviour-Driven Development (BDD), it is the behaviour of the software that the user will experience. Common to all these approaches is that the tests are written before the code is developed, i.e., they are test-first approaches. The approaches are usually better known by their acronyms. This subsection describes these test-first approaches and information on how to apply them is contained in section 3.3.

Test-Driven Development was the first of these approaches to appear. It was introduced as one of the practices within Extreme Programming (XP) back in 1990. It has been practiced for two decades and has been adopted by many software developers, in Agile and traditional projects. However, it is also a good example of an Agile practice that is not used in all projects. One limitation with TDD is that if the developer misunderstands what the software is to do, the unit tests will also include the same misunderstandings, giving passing results even though the software is not working properly. There is some controversy over whether TDD delivers the benefits it promises. Some, such as Jim Coplien, even suggest that unit testing is mostly waste.

TDD is mostly for unit testing by developers.  Agile teams soon came up with the question: What if we could have a way to get the benefits of test-first development for acceptance tests and higher level testing in general? And thus Acceptance Test-Driven Development was born.  (There are also other names for similar higher-level test-first methods; for example, Specification by Example (SBE) from Gojko Adzic.) Later, Dan North wanted to emphasize the behaviours from a business perspective, leading him to give his technique the name Behaviour-Driven Development. ATDD and BDD are in practice very similar concepts. 

Let’s look at these three test-first techniques, TDD, ATDD, and BDD, more closely in the following subsections.

 

Continue reading →

Engineering Quality for Bananas

By Rex Black, Daniel Derr, Michael Tyszkiewicz

[How can dumb monkeys built from free tools help you? Give this article a read to see a case study.  Originally published in Software Testing Professional magazine in 2008, these ideas and techniques are still relevant to SDETs, Technical Test Engineers, and Technical Test Analysts looking to build their own automation solutions using open-source components.]

Arrowhead Electronic Healthcare has been creating eDiarys on handheld devices since 1999. Arrowhead helps pharmaceutical research and marketing organizations document important information about how their products are being used in patients’ homes.

ePRO-LOG is Arrowhead’s third generation eDiary product. The primary design goal of ePRO-LOG is to be able to rapidly deploy diaries used for data collection in clinical trails and disease management programs.

A typical diary may include 100 forms translated in 15 or more languages, and used in several locales. This results in a large number of software builds and configurations.  As a result, we needed an automated test tool to address potential risks and to automate common tasks.

The most important quality risks we wanted to address were:

  • Reliability
  • Translation completeness
  • Functionality of UI
  • Input error checking
  • Verification of requirements

We needed an automated test tool with the following capabilities and features:

  • Address defined risks
  • Produce accurate form-flow diagrams
  • Reduce tedium and opportunity for error in manual testing
  • Save effort associated with manual testing for these risks
  • Improve time-to-market by reducing test cycle duration through 24x7 testing
  • Provide auditable documentation
  • Handle any screen flow or translation without custom test scripts (i.e., be trial-independent)
  • Be easy to implement and cost effective

This is a case study in how we reduced our risks and achieved our test automation objectives in just a few months on a total tools outlay of $0.

Continue reading →

Webinars

No Testing Schools

Recorded May 22, 2014

Agile Testing in the Real World

Recorded December 10, 2013

Profiles of Failure

Recorded April 24, 2013

Agile Testing in the Real World

Recorded February 26, 2013

Podcast Episodes

Five Testing Best Practices 1/11/2010


Length: 1h 27m 53s

A best practice is an approach to doing something that generally gives good results when applied appropriately and thoughtfully. In other words, a best practice is like a good habit. For testing, best practices include analytical risk-based testing strategies, clear test objectives, continuous test process improvement, trained and certified test teams, and intelligently distributed testing work. Based on 25 years of industry experience, Rex Black will talk about what has worked for him, his RBCS associates, and for RBCS clients, over and over again, around the world and across many different types of projects. Come prepared to pick up some new good habits.

Listen now →

Software Testing: Agile Testing Opportunities 3/21/12


Length: 1h 30m 50s

Every lifecycle affects testing, and Agile is no exception. A number of elements of the Agile methodologies can create opportunities for testing, when properly implemented. In this webinar, we'll discuss how nine characteristics of Agile methodologies can turbo-charge your testing, specifically examining: the obvious and not-so-obvious benefits of automated unit testing; how the re-birth of static code analysis makes removing bugs cheap and easy; the use of code coverage metrics to help quantify confidence; the testing benefits of continuous integration; Agile contributions for automated functional testing; how Agile supports high-quality, lightweight requirements; the benefits of stakeholder review of test conditions; reducing the dreaded test crunch with a sustainable workload; and, how Agile methodologies control technical debt and why that matters for testing. In this free webinar, Rex Black discusses the key testing opportunities created by the Agile approach, so that you can recognize and take advantage of them.

Listen now →

Agile Testing Challenges Four Years Later: 9/20/12


Length: 1h 32m 43s

Earlier this year, Rex gave a webinar on how Agile lifecycles create opportunities for testing. Four years ago, Rex gave a presentation on the testing challenges of Agile, first in Auckland, New Zealand and later at various locations around the world. So, what’s changed in the last four years? In this webinar, Rex will revisit his presentation on Agile testing challenges. How are we, as testing professionals, doing in terms of surmounting these challenges? What remains to be done? How can you recognize and manage these challenges in your Agile projects? In this free webinar, Rex will discuss these points and more, helping you be more effective in Agile projects.

Listen now →

Agile Testing in Real World 02/26/13


Length: 1h 31m 10s

Lots of organizations are adopting Agile. Some of these organizations are finding practical ways to integrating testing into Agile lifecycles. In this presentation, Rex will address challenges and opportunities associated with Agile, different ways of adapting Agile lifecycles and the testing within those lifecycles, and how Agile differs from traditional lifecycles. We’ll examine tools and metrics for Agile projects. We’ll address whether Agile can be used for outsourced projects. We’ll look at different options for organizing test teams on Agile projects, and why only some of these options can work. Rex will offer his thoughts on Agile and quality, and what skills and personalities work best in Agile projects. In this free webinar, Rex will discuss these points and more, giving you a better shot at Agile testing success.

Listen now →

Profiles in Failure: 4/24/13


Length: 1h 27m 23s

You can often learn a lot more from a failed project than you can from one that succeeds. So, in this free webinar, Rex offers detailed case studies of three projects that failed—even though they could have succeeded. Each project was very different, but common themes emerge: earlier testing, better quality throughout the project, tighter collaboration between testers and other stakeholders, and better vendor quality management. Rex will discuss how to you can learn the lessons that could have saved these projects so you can avoid being a profile in failure on your next project.

Listen now →

Webinar: Agile Testing in the Real World: 12/10/13


Length: 1h 35m 37s

Lots of organizations are adopting agile. Some of these organizations are finding practical ways to integrating testing into agile lifecycles. In this presentation, Rex will address challenges and opportunities associated with agile, different ways of adapting agile lifecycles and the testing within those lifecycles, and how agile differs from traditional lifecycles. We’ll examine tools and metrics for agile projects. We’ll address whether agile can be used for outsourced projects. We’ll look at different options for organizing test teams on agile projects, and why only some of these options can work. Rex will offer his thoughts on agile and quality, and what skills and personalities work best in agile projects. In this free webinar, Rex will discuss these points and more, giving you a better shot at agile testing success.

Listen now →

Last year, Rex did a series of tweets over a few months, each one calling out a specific test management, testing, project management, or lifestyle best practice—or worst practice. Now, we’ve collected these practices into a two-part series of webinars. At the beginning of the series, Rex will discuss the concept of best practices and refute the software testing canard that there are no best practices. Next, in each webinar, he’ll describe a dozen or more of these practices, including examples of why it is a best practice—or a worst one. Finally, at the end of the series, Rex will give some ideas on how you can remove worst practices and institute best practices in your organization. Tune in and take part in the discussion. After all, continuously improving your skills and knowledge is one of the best practices!

Listen now →

Last year, Rex did a series of tweets over a few months, each one calling out a specific test management, testing, project management, or lifestyle best practice—or worst practice. Now, we’ve collected these practices into a two-part series of webinars. At the beginning of the series, Rex will discuss the concept of best practices and refute the software testing canard that there are no best practices. Next, in each webinar, he’ll describe a dozen or more of these practices, including examples of why it is a best practice—or a worst one. Finally, at the end of the series, Rex will give some ideas on how you can remove worst practices and institute best practices in your organization. Tune in and take part in the discussion. After all, continuously improving your skills and knowledge is one of the best practices!

Listen now →

Webinar: Strategies of Testing, Not Schools: 9/4/14


Length: 1h 14m 12s

A relatively small but vocal assemblage of testers talk about “schools of testing,” but do you have to belong to one? Or can you pick and choose appropriate testing strategies based on your project, organization, and team needs and priorities? In this webinar, Rex will demonstrate how the “schools of testing” concept has actually limited our profession. In the place of this failed paradigm, Rex describes a set of powerful testing strategies that can be customized and blended to create an effective, efficient, and fulfilling test approach. As always, Rex will take your questions after the webinar is over.

Listen now →

Is the V model dead, crushed by the onset of agile methods? In the age of agile, are those who talk about the V model anachronistic dinosaurs? Or was the V model actually ahead of its time, illuminating best practices that now form the core of agile testing? In this talk, Rex will show how the fundamental principles of the V model can be successfully applied in the context of an agile development effort, and how the application of those principles make agile software development even more agile. Sure to stir some controversy, attend Rex’s webinar, submit questions and comments, and make up your own mind.

Listen now →

Webinar: Agile Risk-based Testing: A How-to Guide 11/12/14


Length: 1h 35m 53s

Perhaps you, like many software testers and test managers, are enjoying the benefits of incorporating a lightweight analytical risk-based testing strategy such as RBCS’s Pragmatic Risk Analysis and Management (PRAM) technique into your test approach. What if your organization is adopting an agile methodology? Can you still use PRAM and other similar analytical risk-based testing strategies? Absolutely. In this webinar, Rex will explain the process for integrating quality risk identification and assessment into the release and iteration planning processes, and for integrating quality risk mitigation into each iteration. You’ll return to work with a proven method for injecting risk-based testing into your agile processes.

Listen now →

Why Does Software Quality (Still) Suck


Length: 4h 20m 0s

Software quality, for the most part, sucks. It still sucks, seventy-five years since the advent of the programmable computer. Software bugs are a constant fact of life, thanks to the ubiquity of software and the ubiquity of software bugs. Sometimes the bugs costs millions of dollars or kill people. Why is the reaction so muted? Rather than just accept software bugs as unavoidable, let’s ask the obvious question: Given that manufacturing is able to achieve six sigma levels of quality—i.e., only three defective items per million manufactured—why does software quality still suck? In this webinar, Rex will address some of the real barriers to achieving six sigma quality in software, while at the same time holding software engineering as a profession accountable for not doing nearly as much as we can.

Listen now →

Let’s suppose you bought a car. Six days later, someone from the dealership let himself into your garage, removed the tires on the car, installed some “updated” tires that actually had holes in them, and then left. In the morning, your car was there in the garage, all sad and undriveable on its flat, flabby tires. That’s clearly unacceptable, in fact even criminal, but we allow the same thing to happen all the time with software. Why? In this webinar, Rex will catalog infamous automated software updates, released without sufficient testing to wreak havoc, or at least inconvenience. He’ll then give a detailed roadmap for reducing your chances of being part of the problem.

Listen now →

Some people use the terms “verification” and “validation” interchangeably, but there are significant differences between them. Some people disparage verification, or deny that it’s even involved in testing. However, you can’t adequately build confidence and reduce risk in the software you test without using the proper mix of both. In this webinar, Rex will clarify the meaning of these two terms, give examples, and explain why both are essential to proper software testing.

Listen now →

Webinar: Two Bug Metrics, Millions in Process Improvement


Length: 1h 20m 50s

When we do assessments, we always try to look at process metrics. In most cases, we can find millions of dollars in process improvement opportunities. In this webinar, Rex will show you how two very simple bug metrics, calculated using only two simple facts for each bug report using simple, free spreadsheets you can get from our website, can reveal millions and millions of dollars in potential process improvements. All the more reason to track those bugs! To paraphrase Timothy Leary: Tune in, download, and drop software co

Listen now →

Webinar - Agile Testing in the Real World (Update)


Length: 0h 41m 37s

A majority of software development organizations have adopted Agile methods, but testing in Agile projects remains a major challenge. However, some of these organizations have found practical ways to integrating testing into Agile lifecycles.  In this webinar, Rex will address challenges and opportunities associated with Agile, different ways of adapting Agile lifecycles and the testing within those lifecycles, and how Agile differs from traditional lifecycles.  We’ll examine tools and metrics for Agile projects.  We’ll address whether Agile can be used for outsourced projects.  We’ll look at different options for organizing test teams on Agile projects, and why only some of these options can work.  Rex will offer his thoughts on Agile and quality, and what skills and personalities work best in Agile projects. In this updated free webinar, Rex will discuss these points and more, giving you a better shot at Agile testing success.

Listen now →

One Key Idea: Pairwise Testing Using ACTS 2/16/17


Length: 0h 38m 45s

If you’ve been testing for any length of time, you know that the number of possible test cases is enormous if you try to test all possible combinations of inputs, configuration values, types of data, and so forth. It’s like the mythical monster, the many-headed Hydra, which would sprout two or more new heads for each head that was cut off. Two simple approaches to dealing with combinatorial explosions such as this are equivalence partitioning and boundary value analysis, but those techniques don’t check for interactions between factors. A reasonable, manageable way to test combinations is called pairwise testing, but to do it you’ll need a tool.  In this inaugural One Key Idea session, Rex will demonstrate the use of a free tool, ACTS, built by the US NIST and available for download worldwide. We can’t promise to turn you into Hercules, but you will definitely walk away able to slay the combinatorial Hydra.

Listen now →

One Key Idea: ISTQB Agile Program


Length: 0h 22m 26s

Agile methods have become widespread over the last 20 years, but it’s taken a while for testing to catch up.  However, Agile testing best practices have emerged, and the ISTQB Agile Working Group has taken on the task of capturing those for you.  In this brief webinar, Rex Black, Chair of the ISTQB Agile Working Group, will explain where the program has been, where it is now, and how it is evolving to support Agile testers in their careers. In twenty minutes or less, you’ll learn how the ISTQB Agile syllabi can help you advance your career as an Agile tester.

Listen now →

Stupid Metrics Tricks and How To Avoid Them


Length: 1h 1m 48s

If you have been in software engineering for a while—or in fact just in the working world in general for a while—you’ve probably seen someone do something stupid with metrics. Such mistakes raise a whole bunch of interesting questions. What are the most common metrics mistakes? Why are they mistakes? Why do people make these mistakes? Are you making these mistakes? Why use metrics at all, when there are so many mistakes? In this talk, Rex will give real-world examples of these mistakes, explain the management and economic theories behind metrics, and help you find ways to implement metrics that aren’t stupid.

Listen now →

Training

ISTQB Virtual Advanced Security Tester Boot Camp

The Advanced Security Tester Boot Camp course, created by Rex Black, past President of the International Software Testing Qualifications Board (ISTQB), past President of the American Software Testing Qualifications Board (ASTQB) and co-author of a number of International Software Testing Qualifications Board syllabi, is ideal for testers and test teams preparing for certification in a short timeframe with time and money constraints.

View details →

ISTQB Virtual Advanced Test Automation Engineer Boot Camp

The Advanced Test Automation Engineer Boot Camp, created by Rex Black, past President of the International Software Testing Qualifications Board (ISTQB), past President of the American Software Testing Qualifications Board (ASTQB) and co-author of a number of International Software Testing Qualifications Board syllabi, is ideal for testers and test teams preparing for certification in a short timeframe with time and money constraints.

View details →

Virtual Foundation Business Analyst Boot Camp

The Foundation Business Analyst Boot Camp, created by Rex Black, past President of the International Software Testing Qualifications Board (ISTQB), past President of the American Software Testing Qualifications Board (ASTQB) and co-author of a number of International Software Testing Qualifications Board syllabi, is ideal for testers and test teams preparing for certification in a short timeframe with time and money constraints.

View details →

Black-box Bug-a-thon

If you are looking to spend one day learning powerful test techniques in a hands-on way using real apps, and having fun while doing so in a friendly yet competitive setting, this black-box bug-a-thon is for you.

View details →


Copyright ® 2017 Rex Black Consulting Services.
All Rights Reserved.
ISTQB Logo ASTQB Logo IREB Logo PMI Logo ISTQB Logo
PMI is a registered mark of the Project Management Institute, Inc.