Resources

Mistakes lead to the introduction of defects (also called bugs). As I personally am aware, like all human beings I can make mistakes at any point in time, no matter what I might be working on. So it is on projects, where the business analyst can put a defect into a requirements specification, a tester can put a defect into a test case, a programmer can put a defect into the code, a technical writer can put a defect into the user guide, and so forth. Any work product can and often will have defects because any worker can and will make mistakes!

Now, I like to drink good wine, and I’ve collected bottles from around the world on my travels. Some I’ve stashed away for years, waiting for the right time to drink them, which always comes sooner or later. Some of these bottles have gotten a lot more valuable over the years. Odd as this will sound, in just a few ways, bugs are like wine. They definitely get more expensive with age, taking more effort and thus incurring more cost the longer they are in the system. Also, sooner or later most bugs need to be fixed. However, it’s definitely not a good idea to leave bugs lying around—or perhaps crawling around—in a cellar, and they’re certainly not appetizing!

Continue reading →

To celebrate completion of the update to Advanced Software Testing: Volume I, here is an excerpt of Chapter 3. This is the central chapter of the book, addressing test design techniques.

We start with the most basic of specification-based test design techniques, equivalence partitioning.

Conceptually, equivalence partitioning is about testing various groups that are expected to be handled the same way by the system and exhibit similar behavior.  Those groups can be inputs, outputs, internal values, calculations, or time values, and should include valid and invalid groups. We select a single value from each equivalence partition, and this allows us to reduce the number of tests.  We can calculate coverage by dividing the number of equivalence partitions tested by the number identified, though generally the goal is to achieve 100% coverage by selecting at least one value from each partition.

This technique is universally applicable at any test level, in any situation where we can identify the equivalence partitions. Ideally, those partitions are independent, though some amount of interaction between input values does not preclude the use of the technique. This technique is also very useful in constructing smoke tests, though testing of some of the less-risky partitions frequently is omitted in smoke tests. This technique will find primarily functional defects where data is processed improperly in one or more partitions. The key to this technique is to take care that the values in each equivalence partition are indeed handled the same way; otherwise, you will miss potentially important test values.

Continue reading →

Since it is not possible to test everything, it is necessary to pick a subset of the overall set of tests to be run. Read this article to discover how quality risks analysis can help one focus the test effort.

Continue reading →

"Do more with less. Work smarter not harder. Same coverage, fewer testers." If you're like a lot of testers and test managers, you'll be hearing statements like those a lot in 2009, since we appear headed for another tight economic period. If you need a way to demonstrate quick, measurable efficiency gains in your test operation, read this  short article to learn four great ideas that will help you improve your software test efficiency. 

Continue reading →

This software testing article is excerpted from Chapter 10 of Rex Black's upcoming book Managing the Testing Process, 3e.  Over the last twenty years, outsource development of one or more key components in the system has come to dominate software and hardware systems engineering. The trend started in hardware in the 1990s. RBCS clients like Dell, Hitachi, Hewlett Packard, and other computer systems vendors took advantage of cheap yet educated labor overseas to compete effectively in an increasingly commoditized market. By the end of 2002, three years into a spectacular IT downturn that saw computer science enrollments in the United States fall to less than half of their 1999 levels, price had become the primary determinant in most IT project decisions.  Mass outsourcing of software projects took hold, and it continues unabated to this day... Read this article to get a picture of the effects of outsourcing software testing.

Continue reading →

You're familiar with test automation, but what is a dumb monkey? How can it help you automate your testing and explore very large screen flows and input sets? Is it true that you can build dumb monkeys from freeware with no tool budget? What kind of quality risks can dumb monkeys address? Read this article to learn the answers to these and other test automation questions.

Continue reading →

Continue reading →

Continue reading →

This is an excerpt from my book, Expert Test Manager, written with James Rommens and Leo van der Aalst. I hope it helps you think more clearly about the test strategies you use.

A test policy contains the mission and objectives of testing along with metrics and goals associated with the effectiveness, efficiency, and satisfaction with which we achieve those objectives. In short, the policy defines why we test. While it might also include some high-level description of the fundamental test process, in general the test policy does not talk about how we test.

The document that describes how we test is the test strategy. In the test strategy, the test group explains how the test policy will be implemented. This document should be a general description that spans multiple projects. While the test strategy can describe how testing is done for all projects, organizations might choose to have separate documents for various types of projects. For example, an organization might have a sequential lifecycle test strategy, an Agile test strategy, and a maintenance test strategy.

Continue reading →

[Note: This is an excerpt from Agile Testing Foundations: An ISTQB Foundation Level Agile Tester Guide, by Rex Black, Marie Walsh, Gerry Coleman, Bertrand Cornanguer, Istvan Forgacs, Kari Kakkonen, and Jan Sabak, published July 2017. Kari Kakkonen wrote this selection. The authors are all members of the ISTQB Working Group that wrote the ISTQB Agile Tester Foundation syllabus.]

The traditional way of developing code is to write the code first, and then test it. Some of the major challenges of this approach are that testing is generally conducted late in the process and it is difficult to achieve adequate test coverage. Test-first practices can help solve these challenges. In this environment, tests are designed first, in a collaboration between business stakeholders, testers, and developers.  Their knowledge of what will be tested helps developers write code that fulfils the tests. A test-first approach allows the team to focus on and clarify the expressed needs through a discussion of how to test the resulting code. Developers can use these tests to guide their development.  Developers, testers, and business stakeholders can use these tests to verify the code once it is developed.

A number of test-first practices have been created for Agile projects, as mentioned in section 2.1 of this book. They tend to be called X Driven Development, where X stands for the driving force for the development. In Test Driven Development (TDD), the driving force is testing. In Acceptance Test-Driven Development (ATDD), it is the acceptance tests that will verify the implemented user story. In Behaviour-Driven Development (BDD), it is the behaviour of the software that the user will experience. Common to all these approaches is that the tests are written before the code is developed, i.e., they are test-first approaches. The approaches are usually better known by their acronyms. This subsection describes these test-first approaches and information on how to apply them is contained in section 3.3.

Test-Driven Development was the first of these approaches to appear. It was introduced as one of the practices within Extreme Programming (XP) back in 1990. It has been practiced for two decades and has been adopted by many software developers, in Agile and traditional projects. However, it is also a good example of an Agile practice that is not used in all projects. One limitation with TDD is that if the developer misunderstands what the software is to do, the unit tests will also include the same misunderstandings, giving passing results even though the software is not working properly. There is some controversy over whether TDD delivers the benefits it promises. Some, such as Jim Coplien, even suggest that unit testing is mostly waste.

TDD is mostly for unit testing by developers.  Agile teams soon came up with the question: What if we could have a way to get the benefits of test-first development for acceptance tests and higher level testing in general? And thus Acceptance Test-Driven Development was born.  (There are also other names for similar higher-level test-first methods; for example, Specification by Example (SBE) from Gojko Adzic.) Later, Dan North wanted to emphasize the behaviours from a business perspective, leading him to give his technique the name Behaviour-Driven Development. ATDD and BDD are in practice very similar concepts. 

Let’s look at these three test-first techniques, TDD, ATDD, and BDD, more closely in the following subsections.

Continue reading →

Let’s suppose you bought a car. Six days later, someone from the dealership let himself into your garage, removed the tires on the car, installed some “updated” tires that actually had holes in them, and then left. In the morning, your car was there in the garage, all sad and undriveable on its flat, flabby tires. That’s clearly unacceptable, in fact even criminal, but we allow the same thing to happen all the time with software. Why? In this webinar, Rex will catalog infamous automated software updates, released without sufficient testing to wreak havoc, or at least inconvenience. He’ll then give a detailed roadmap for reducing your chances of being part of the problem.

Listen now →

If you’ve been testing for any length of time, you know that the number of possible test cases is enormous if you try to test all possible combinations of inputs, configuration values, types of data, and so forth. It’s like the mythical monster, the many-headed Hydra, which would sprout two or more new heads for each head that was cut off. Two simple approaches to dealing with combinatorial explosions such as this are equivalence partitioning and boundary value analysis, but those techniques don’t check for interactions between factors. A reasonable, manageable way to test combinations is called pairwise testing, but to do it you’ll need a tool.  In this inaugural One Key Idea session, Rex will demonstrate the use of a free tool, ACTS, built by the US NIST and available for download worldwide. We can’t promise to turn you into Hercules, but you will definitely walk away able to slay the combinatorial Hydra.

Listen now →

If you are testing a simple mobile app, you may find it relatively easy to find representative test data. However, what if you are testing enterprise scale applications? In the enterprise data center, one hundred or more applications of various sizes, complexity, and criticality co-exist, operating on various data repositories, in some cases shared data repositories. In some cases, disparate data repositories hold related data, and the ability to test integration across applications that access these data sets is critical.  In this keynote speech, Rex Black will talk about the challenges facing his clients as they deal with these testing problems. You’ll go away with a better understanding of the nature of the challenges, as well as ideas on how to handle them, grounded in lessons Rex has learned in over 30 years of software engineering and testing.

Listen now →

If you are testing a simple mobile app, you may find it relatively easy to find representative test data. However, what if you are testing enterprise scale applications? In the enterprise data center, one hundred or more applications of various sizes, complexity, and criticality co-exist, operating on various data repositories, in some cases shared data repositories. In some cases, disparate data repositories hold related data, and the ability to test integration across applications that access these data sets is critical.  In this webinar, Rex Black will talk about the challenges facing his clients as they deal with these testing problems. You’ll go away with a better understanding of the nature of the challenges, as well as ideas on how to handle them, grounded in lessons Rex has learned in over 30 years of software engineering and testing.

Listen now →

Shifting defect discovery to earlier activities (Shift Left) reduces rework and improves quality deliveries to customers. A lot of attention is paid to this part of “Shift Left and Down”, but how do we do the Shift Down? Root Cause Analysis (RCA) is the answer. A variety of techniques can be used with varying formalities, cost, and results; ranging from informal, 5-why, Ishikawa/fishbone to Cause-Effect.  Which technique is the right one to use, and when should it be applied? Is your organization ready for RCA? What do you do with the results? These questions and others will be discussed by Ed Weller, who will draw on 20 years of experience using RCA and onsite training across a wide range of businesses.

Listen now →

If you’ve been testing for any length of time, you know that the number of possible test cases is enormous if you try to test all possible combinations of inputs, configuration values, types of data, and so forth. It’s like the mythical monster, the many-headed Hydra, which would sprout two or more new heads for each head that was cut off. Two simple approaches to dealing with combinatorial explosions such as this are equivalence partitioning and boundary value analysis, but those techniques don’t check for interactions between factors. A reasonable, manageable way to test combinations is called pairwise testing, but to do it you’ll need a tool.  In this One Key Idea session, Rex will demonstrate the use of a free tool, ACTS, built by the US NIST and available for download worldwide. We can’t promise to turn you into Hercules, but you will definitely walk away able to slay the combinatorial Hydra.

Listen now →

Listen now →