Our notable partnerships give us access to the best resources and tools for the job.
Get software testing training resources:
Best practices are superior methods or innovative practices that contribute to the improved performance of an organization when applied with appropriate tailoring and under appropriate circumstances. Best practices represent the current state of the art in a given industry.
When I wrote my book Critical Testing Processes in the early 2000s, I started with the premise that some test processes are critical, some are not. I designed this lightweight framework for test process improvement in order to focus the test team and test manager on a few test areas that they simply must do properly. This contrasts with the more expansive and complex models inherent in TPI and TMM. In addition, the Critical Testing Processes (CTP) framework eschews the prescriptive elements of TMM and TPI since it does not impose an arbitrary, staged maturity model.
What’s the problem with prescriptive models? In my consulting work, I have found that businesses want to make improvements based on the business value of the improvement and the organizational pain that improvement will alleviate. A simplistic maturity rating might lead a business to make improvements in parts of the overall software process or test process that are actually less problematic or less important than other parts of the process simply because the model listed them in order.
CTP is a non-prescriptive process model. It describes the important software processes and what should happen in them, but it doesn’t put them in any order of improvement. This makes CTP a very flexible model. It allows you to identify and deal with specific challenges to your test processes. It identifies various attributes of good processes, both quantitative and qualitative. It allows you to use business value and organizational pain to select the order and importance of improvements. It is also adaptable to all software development lifecycle models.
Why Care About Testing Processes?
As a test professional with almost two decades in the computer field, I’m pleased to see a new publication devoted to our field, the Journal of Software Testing Professionals, and honored to serve as a Contributing Editor. As my first task in this role, I have been asked to write and collaborate with other software testing professionals to produce a series of articles addressing what we consider critical processes that the test professional must master in order for her to succeed. In each of these articles, we’ll examine one critical process closely, offering practical advice on what works and caveats on what can cause problems
So, you are responsible for managing a computer hardware or software test project? Congratulations! Maybe you’ve just moved up from test engineering or moved over from another part of the development team, or maybe you’ve been doing test projects for a while. Whether you are a test manager, a development manager, a technical or project leader, or an individual contributor with some level of responsibility for your company’s test and quality assurance program, you’re probably looking for some ideas on how to manage the unique beast that is a test project.
In the last issue, I introduced this series of articles on critical software testing processes. For our first article, let’s look at the processes associated with an essential precondition for any test activities: creation, delivery, and installation of a software release into the test environment. Without software to test, there can be no software testing, but, as obvious as this is, many software development organizations do a poor job of managing test release processes. Often, the role of building a test release devolves into the test organization, which is generally ill-prepared to handle the tasks, and the processes are defined reactively— to put out fires—not proactively—to handle key roles optimally.
In the last issue, I dealt with a collaborative test process, getting test releases into the test lab. In this article, let’s look instead at an internal test process, managing the execution of tests against a test release. One can think of the test manager’s role during the test execution process as one of managing a search for the unexpected. While all managers are expected to manage unexpected events within their areas of responsibility, the test manager is the only manager in the software development organization whose area of responsibility is searching for the unexpected. This search takes the form of running a set of tests. How does a test team take these tests, run them against an installed system, and capture information about the system under test, the test cases, and the test execution process itself?
Pervasive testing means getting the right people working together through the right processes at the right time for high-ROI testing. Pervasive testing is the final piece of the test investment puzzle, and just as important as all the others. Through pervasive testing, all the ideas we’ve explored so far come together.
If you are a software tester, programmer, or manager, you probably know that developing secure software is no longer simply desirable—it’s completely essential.
Some might assume that most security problems arise from the operating system or networking layers, well below the application code they are working on. However, figures for Web-based applications show that over three-quarters of security exploits arose from applications.
So, you know you need secure code, but how to get there? What are your security risks? What security failures and bugs do you have? What do these security risks, failures, and bugs mean? How can you reduce security risk in a way that doesn’t create new problems? How do you monitor my progress over time? This article will outline seven steps that will allow you to answer these and other questions as you improve your software’s security.
This is an excerpt from my book, Expert Test Manager, written with James Rommens and Leo van der Aalst. I hope it helps you think more clearly about the test strategies you use.
A test policy contains the mission and objectives of testing along with metrics and goals associated with the effectiveness, efficiency, and satisfaction with which we achieve those objectives. In short, the policy defines why we test. While it might also include some high-level description of the fundamental test process, in general the test policy does not talk about how we test.
The document that describes how we test is the test strategy. In the test strategy, the test group explains how the test policy will be implemented. This document should be a general description that spans multiple projects. While the test strategy can describe how testing is done for all projects, organizations might choose to have separate documents for various types of projects. For example, an organization might have a sequential lifecycle test strategy, an Agile test strategy, and a maintenance test strategy.
[Note: This is an excerpt from Agile Testing Foundations: An ISTQB Foundation Level Agile Tester Guide, by Rex Black, Marie Walsh, Gerry Coleman, Bertrand Cornanguer, Istvan Forgacs, Kari Kakkonen, and Jan Sabak, published July 2017. Kari Kakkonen wrote this selection. The authors are all members of the ISTQB Working Group that wrote the ISTQB Agile Tester Foundation syllabus.]
The traditional way of developing code is to write the code first, and then test it. Some of the major challenges of this approach are that testing is generally conducted late in the process and it is difficult to achieve adequate test coverage. Test-first practices can help solve these challenges. In this environment, tests are designed first, in a collaboration between business stakeholders, testers, and developers. Their knowledge of what will be tested helps developers write code that fulfils the tests. A test-first approach allows the team to focus on and clarify the expressed needs through a discussion of how to test the resulting code. Developers can use these tests to guide their development. Developers, testers, and business stakeholders can use these tests to verify the code once it is developed.
A number of test-first practices have been created for Agile projects, as mentioned in section 2.1 of this book. They tend to be called X Driven Development, where X stands for the driving force for the development. In Test Driven Development (TDD), the driving force is testing. In Acceptance Test-Driven Development (ATDD), it is the acceptance tests that will verify the implemented user story. In Behaviour-Driven Development (BDD), it is the behaviour of the software that the user will experience. Common to all these approaches is that the tests are written before the code is developed, i.e., they are test-first approaches. The approaches are usually better known by their acronyms. This subsection describes these test-first approaches and information on how to apply them is contained in section 3.3.
Test-Driven Development was the first of these approaches to appear. It was introduced as one of the practices within Extreme Programming (XP) back in 1990. It has been practiced for two decades and has been adopted by many software developers, in Agile and traditional projects. However, it is also a good example of an Agile practice that is not used in all projects. One limitation with TDD is that if the developer misunderstands what the software is to do, the unit tests will also include the same misunderstandings, giving passing results even though the software is not working properly. There is some controversy over whether TDD delivers the benefits it promises. Some, such as Jim Coplien, even suggest that unit testing is mostly waste.
TDD is mostly for unit testing by developers. Agile teams soon came up with the question: What if we could have a way to get the benefits of test-first development for acceptance tests and higher level testing in general? And thus Acceptance Test-Driven Development was born. (There are also other names for similar higher-level test-first methods; for example, Specification by Example (SBE) from Gojko Adzic.) Later, Dan North wanted to emphasize the behaviours from a business perspective, leading him to give his technique the name Behaviour-Driven Development. ATDD and BDD are in practice very similar concepts.
Let’s look at these three test-first techniques, TDD, ATDD, and BDD, more closely in the following subsections.
[How can dumb monkeys built from free tools help you? Give this article a read to see a case study. Originally published in Software Testing Professional magazine in 2008, these ideas and techniques are still relevant to SDETs, Technical Test Engineers, and Technical Test Analysts looking to build their own automation solutions using open-source components.]
Arrowhead Electronic Healthcare has been creating eDiarys on handheld devices since 1999. Arrowhead helps pharmaceutical research and marketing organizations document important information about how their products are being used in patients’ homes.
ePRO-LOG is Arrowhead’s third generation eDiary product. The primary design goal of ePRO-LOG is to be able to rapidly deploy diaries used for data collection in clinical trails and disease management programs.
A typical diary may include 100 forms translated in 15 or more languages, and used in several locales. This results in a large number of software builds and configurations. As a result, we needed an automated test tool to address potential risks and to automate common tasks.
The most important quality risks we wanted to address were:
We needed an automated test tool with the following capabilities and features:
This is a case study in how we reduced our risks and achieved our test automation objectives in just a few months on a total tools outlay of $0.
[The following is an excerpt from my upcoming book, Mobile Testing: An ASTQB-BCS Foundation Guide, coming out in summer 2018. This section talks about the important issue of mobile test environments.]
Earlier in this book, I gave the example of my client that gave iPhones to all the sales associates in their stores. Because of that, they have a fairly homogeneous environment. This is true not only of the client-side, but also the connectivity in the stores (through the in-store Wi-Fi installed by them) and the back-end servers in the stores and in their data centers.
In terms of mobile testing, this homogeneity makes them the exception that proves the rule, as the saying goes. They don’t have to deal with widespread and unpredictable environmental diversity. What diversity they do have is due to deliberate decisions. For example, they have to depreciate the iPhones, so over time there came to be multiple generations of iPhones in stores.
Recorded April 11, 2019
Recorded March 19, 2019
Recorded December 18, 2018
Recorded October 8, 2018
Recorded August 16, 2018
Recorded July 17, 2018
Recorded May 3, 2018
Recorded March 26, 2018
Recorded February 26, 2018
Recorded December 5, 2017
Recorded November 28, 2017
Recorded October 9, 2017
Recorded October 9, 2017
Recorded February 16, 2017
Recorded December 28, 2016
Recorded October 24, 2016
Recorded September 7, 2016
Recorded May 19, 2016
Recorded April 4, 2016
Recorded March 30, 2016
Recorded March 3, 2016
Recorded October 8, 2014
Recorded September 5, 2014
Recorded July 14, 2014
Recorded June 17, 2014
Recorded May 22, 2014
Recorded December 10, 2013
Recorded April 24, 2013
Recorded February 26, 2013
Recorded November 7, 2012
Recorded September 20, 2012
Recorded March 21, 2012
Recorded September 3, 2009
Length: 1h 49m 0s
Software quality, for the most part, sucks. It still sucks, seventy-five years since the advent of the programmable computer. Software bugs are a constant fact of life, thanks to the ubiquity of software and the ubiquity of software bugs. Sometimes the bugs costs millions of dollars or kill people. Why is the reaction so muted? Rather than just accept software bugs as unavoidable, let’s ask the obvious question: Given that manufacturing is able to achieve six sigma levels of quality—i.e., only three defective items per million manufactured—why does software quality still suck? In this webinar, Rex will address some of the real barriers to achieving six sigma quality in software, while at the same time holding software engineering as a profession accountable for not doing nearly as much as we can.
Length: 1h 30m 0s
Let’s suppose you bought a car. Six days later, someone from the dealership let himself into your garage, removed the tires on the car, installed some “updated” tires that actually had holes in them, and then left. In the morning, your car was there in the garage, all sad and undriveable on its flat, flabby tires. That’s clearly unacceptable, in fact even criminal, but we allow the same thing to happen all the time with software. Why? In this webinar, Rex will catalog infamous automated software updates, released without sufficient testing to wreak havoc, or at least inconvenience. He’ll then give a detailed roadmap for reducing your chances of being part of the problem.
Some people use the terms “verification” and “validation” interchangeably, but there are significant differences between them. Some people disparage verification, or deny that it’s even involved in testing. However, you can’t adequately build confidence and reduce risk in the software you test without using the proper mix of both. In this webinar, Rex will clarify the meaning of these two terms, give examples, and explain why both are essential to proper software testing.
Length: 1h 20m 50s
When we do assessments, we always try to look at process metrics. In most cases, we can find millions of dollars in process improvement opportunities. In this webinar, Rex will show you how two very simple bug metrics, calculated using only two simple facts for each bug report using simple, free spreadsheets you can get from our website, can reveal millions and millions of dollars in potential process improvements. All the more reason to track those bugs! To paraphrase Timothy Leary: Tune in, download, and drop software co
Length: 0h 41m 37s
A majority of software development organizations have adopted Agile methods, but testing in Agile projects remains a major challenge. However, some of these organizations have found practical ways to integrating testing into Agile lifecycles. In this webinar, Rex will address challenges and opportunities associated with Agile, different ways of adapting Agile lifecycles and the testing within those lifecycles, and how Agile differs from traditional lifecycles. We’ll examine tools and metrics for Agile projects. We’ll address whether Agile can be used for outsourced projects. We’ll look at different options for organizing test teams on Agile projects, and why only some of these options can work. Rex will offer his thoughts on Agile and quality, and what skills and personalities work best in Agile projects. In this updated free webinar, Rex will discuss these points and more, giving you a better shot at Agile testing success.
Length: 0h 38m 45s
If you’ve been testing for any length of time, you know that the number of possible test cases is enormous if you try to test all possible combinations of inputs, configuration values, types of data, and so forth. It’s like the mythical monster, the many-headed Hydra, which would sprout two or more new heads for each head that was cut off. Two simple approaches to dealing with combinatorial explosions such as this are equivalence partitioning and boundary value analysis, but those techniques don’t check for interactions between factors. A reasonable, manageable way to test combinations is called pairwise testing, but to do it you’ll need a tool. In this inaugural One Key Idea session, Rex will demonstrate the use of a free tool, ACTS, built by the US NIST and available for download worldwide. We can’t promise to turn you into Hercules, but you will definitely walk away able to slay the combinatorial Hydra.
Length: 0h 22m 26s
Agile methods have become widespread over the last 20 years, but it’s taken a while for testing to catch up. However, Agile testing best practices have emerged, and the ISTQB Agile Working Group has taken on the task of capturing those for you. In this brief webinar, Rex Black, Chair of the ISTQB Agile Working Group, will explain where the program has been, where it is now, and how it is evolving to support Agile testers in their careers. In twenty minutes or less, you’ll learn how the ISTQB Agile syllabi can help you advance your career as an Agile tester.
Length: 1h 1m 48s
If you have been in software engineering for a while—or in fact just in the working world in general for a while—you’ve probably seen someone do something stupid with metrics. Such mistakes raise a whole bunch of interesting questions. What are the most common metrics mistakes? Why are they mistakes? Why do people make these mistakes? Are you making these mistakes? Why use metrics at all, when there are so many mistakes? In this talk, Rex will give real-world examples of these mistakes, explain the management and economic theories behind metrics, and help you find ways to implement metrics that aren’t stupid.
Length: 1h 0m 28s
Shift left. Continuous integration and continuous delivery (CI/CD). Continuous deployment. DevOps. What is all this stuff and what does it mean for you as the tester? In this keynote, Rex Black will explain these concepts and their test implications. He’ll then describe the emerging role of the SDET (Software Development Engineering in Test, also called SET) and what SDETs do. Yes, being an SDET is about test automation, but it’s about a lot more than that, and Rex will give you some examples of things you can expect to do as an SDET in a shift left world over the coming decade. Don’t worry. Life as a tester in the SDET reality is gonna be fun and exciting, and Rex will give you some ideas how.
Length: 0h 18m 11s
Test automation is all the rage. Spinning away in Agile lifecycles or playing key roles in DevOps pipelines, automation is supposed to be everywhere, right? However, such widespread automation is a big investment. If you want to obtain management approval for the kind of automation investments all the webinar and conference talking heads are saying you simply must do right now, you better be able to talk automation ROI. In this One Key Idea session, Rex will explain the measurable business benefits of test automation and how to calculate automation ROI. In twenty minutes or less, you’ll learn how to bridge the gap between automation techno-speak and the managerial bottom-line focus.
Length: 0h 38m 0s
For nearly ten years, RBCS has run a highly successful free webinar series. In 2018, we’re adding the Two Points of View at Two series to our monthly webinar rotation. In each of these sessions, Rex Black will talk with another software luminary about topics of mutual interest, where the two have some different views, and then Rex opens the floor to questions.
In this inaugural session, Rex is happy to welcome Maaret Pyhäjärvi . Maaret’s bio describes her as feedback fairy with a day job at F-Secure, where they call her a Lead Quality Engineer. She identifies as empirical technologist, tester and programmer, catalyst for improvement, author and speaker, and community facilitator and conference organizer. You can catch her latest thoughts on her blog at http://visible-quality.blogspot.fi
In this session, Rex and Maaret will discuss tester-developer collaboration and the relationships between testers and developers. How to approach developers for collaboration? How do testers-developers ratios affect relationships? What about people who move between tester and developer roles? Join Rex and Maaret to hear their thoughts and ask your questions.
Length: 0h 30m 44s
Technical debt is bad. It’s smart to carefully manage technical debt. Defects are a form of technical debt. Do you know how to measure how well you are managing defect-related technical debt? In this One Key Idea session, Rex will demonstrate two simple defect metrics, easily extractable from any defect management tool, which can give you useful insights into what’s happening with defect-related technical debt. In twenty minutes or less, you’ll learn what these metrics can tell you and how you can use them to manage your technical debt better.
Length: 0h 30m 16s
Hiring and managing distributed, international test teams
Phil Lew, President of XBOSoft, an international software testing firm, joins Rex Black to discuss the critical topics of hiring and managing teams of testers who work around the world. How do distance, culture, and language create challenges, and how have Phil and Rex dealt with those challenges in the past? You won’t want to miss Phil and Rex’s points of view on these important topics.
Length: 1h 14m 56s
What should testing accomplish and how do we recognize success? How should we approach testing? What specific activities need to be planned for each release, each project, each iteration? What risks can affect testing, and how can we manage those risks? How does lifecycle affect these documents? These are critical questions for testers and test managers, yet often they go unanswered or have answers that aren’t fully thought through. In this webinar, based on years of experience helping testers and test teams optimize their testing processes, Rex will discuss test policies, strategies and plans. Join this webinar, illustrated with examples throughout, to learn concepts and ideas that you can apply to your testing processes right away.
Length: 0h 30m 54s
In this month’s “Two Points of View at Two” session, Rex is happy to welcome Nivia S. Henry. Nivia fundamentally believes that happy people, working in a healthy environment, will do great things. This philosophy has driven her to build a 15+ year career creating and supporting high-performing teams. Her career path has included agile coaching, enterprise agile transformations, product management, and people leadership. Today, Nivia applies her hard-earned experience as an Quality and Web Engineering Manager at Spotify. In this session, she'll discuss Spotify's perspective on Quality and focus on how the organization has organically grown its test automation practice.
Length: 0h 56m 31s
All too often, people believe that they have to run all their system tests through the GUI, and that includes automated tests. However, it is possible to run automated system tests through a variety of interfaces, such as command lines, APIs, data layers, network services, and more. In this webinar, Rex will give two such examples of sophisticated automated system test platforms capable of quickly running thousands of tests with very low false positive rates, flaky test rates, and test maintenance rates. One tested a data layer interface, the other a network services interface. Each used a flexible and maintainable keyword driven architecture. Come ready to open your mind to new ways of automating system testing, and leave with ideas you can apply to automating your tests away from the GUI.
Length: 0h 27m 3s
If you are a hiring manager, you might already know that a bad hiring decision is one of the biggest mistakes you can make. One easy mistake to make is hiring someone who tells a convincing story about their abilities, but can’t actually do the work. How can you separate the glib talker from the accomplished doer? Well, in this One Key Idea session, Rex will explain how to use audition interviews, illustrating the concept with case studies. In twenty minutes or less, you’ll gain a powerful tool for making smarter hiring decisions.
Two Points of View at Two: Short High-impact Talks about Testing, with Michael Mah
In this session, Rex is happy to welcome Michael Mah. As managing partner of QSM Associates for over 20 years, Michael and his colleagues have assembled one of the largest worldwide databases containing software quality and productivity trends from over 10,000 completed projects. Michael and Rex will discuss defects, or bugs - and how in today’s competitive market, they can either make or break your company. We’ll discuss real world case studies and key findings from real world benchmarking and diagnostic engagements on what separates the leaders from has-beens. Michael and Rex have over 60 years of collective experience dealing with defects and their consequences, so they will discuss these important topics, and take your questions.
Length: 1h 3m 36s
Shifting defect discovery to earlier activities (Shift Left) reduces rework and improves quality deliveries to customers. A lot of attention is paid to this part of “Shift Left and Down”, but how do we do the Shift Down? Root Cause Analysis (RCA) is the answer. A variety of techniques can be used with varying formalities, cost, and results; ranging from informal, 5-why, Ishikawa/fishbone to Cause-Effect. Which technique is the right one to use, and when should it be applied? Is your organization ready for RCA? What do you do with the results? These questions and others will be discussed by Ed Weller, who will draw on 20 years of experience using RCA and onsite training across a wide range of businesses.
The Advanced Security Tester Boot Camp course, created by Rex Black, past President of the International Software Testing Qualifications Board (ISTQB), past President of the American Software Testing Qualifications Board (ASTQB) and co-author of a number of International Software Testing Qualifications Board syllabi, is ideal for testers and test teams preparing for certification in a short timeframe with time and money constraints.
The Advanced Test Automation Engineer Boot Camp, created by Rex Black, past President of the International Software Testing Qualifications Board (ISTQB), past President of the American Software Testing Qualifications Board (ASTQB) and co-author of a number of International Software Testing Qualifications Board syllabi, is ideal for testers and test teams preparing for certification in a short timeframe with time and money constraints.
The Foundation Business Analyst Boot Camp, created by Rex Black, past President of the International Software Testing Qualifications Board (ISTQB), past President of the American Software Testing Qualifications Board (ASTQB) and co-author of a number of International Software Testing Qualifications Board syllabi, is ideal for testers and test teams preparing for certification in a short timeframe with time and money constraints.