Resources

Our notable partnerships give us access to the best resources and tools for the job.

Get software testing training resources:

Test management is the planning, estimating, monitoring and control of test activities, typically carried out by a test manager.


Resources for “Management”

Articles

More and more projects involve more integration of custom developed or commercial-off-the-shelf (COTS) components, rather than in-house development or enhancement of software. In effect, these two approaches constitute direct or indirect outsourcing of some or all of the development work for a system, respectively. While some project managers see such outsourcing of development as reducing the overall risk, each integrated component can bring with it significantly increased risks to system quality. Read this software testing article to learn about the factors that lead to these risks ,and strategies you can use to manage them.

Continue reading →

Empirix's QAZone with Rex Black

By Marina Gil Santamaria

From certification to automation, expert thoughts on where the testing industry is and where it's headed.

Continue reading →

Recently, we worked on a high-risk, high-visibility system where performance testing ("Let's just make sure it handles the load") was the last item on the agenda. As luck would have it, the system didn't handle the load, and very long days and nights ensued. Why it does have to be this way... Read this article about Risk Mitigation to ensure this doesn’t happen to you.

Continue reading →

"Do more with less. Work smarter not harder. Same coverage, fewer testers." If you're like a lot of testers and test managers, you'll be hearing statements like those a lot in 2009, since we appear headed for another tight economic period. If you need a way to demonstrate quick, measurable efficiency gains in your test operation, read this  short article to learn four great ideas that will help you improve your software test efficiency. 

Continue reading →

A Case Study in Successful Risk-Based Testing at CA

By Rex Black, Peter Nash and Ken Young

This article presents a case study of a risk-based testing pilot project at CA, the world's leading independent IT management software company. The development team chosen for this pilot is responsible for a widely-used mainframe software product called CA SYSVIEWR Performance Management, an intuitive tool for proactive management and real-time monitoring of z/OS environments.  By analyzing a vast array of performance metrics, CA SYSVIEW can help organizations identify and resolve problems quickly.

CA piloted risk-based testing as part of our larger effort to ensure the quality of the solutions we deliver.  The pilot consisted of six main activities:

  • Training key stakeholders on risk-based testing
  • Holding a quality risk analysis session
  • Analyzing and refining the quality risk analysis
  • Aligning the testing with the quality risks
  • Guiding the testing based on risks
  • Assessing benefits and lessons

This article addresses each of these areas - as well as some of the broader issues associated with risk-based testing.  Click here to read the version of this software testing article as published in Better Software Testing.

Continue reading →

When I wrote my book Critical Testing Processes in the early 2000s, I started with the premise that some test processes are critical, some are not. I designed this lightweight framework for test process improvement in order to focus the test team and test manager on a few test areas that they simply must do properly. This contrasts with the more expansive and complex models inherent in TPI and TMM.  In addition, the Critical Testing Processes (CTP) framework eschews the prescriptive elements of TMM and TPI since it does not impose an arbitrary, staged maturity model.

What’s the problem with prescriptive models?  In my consulting work, I have found that businesses want to make improvements based on the business value of the improvement and the organizational pain that improvement will alleviate. A simplistic maturity rating might lead a business to make improvements in parts of the overall software process or test process that are actually less problematic or less important than other parts of the process simply because the model listed them in order.

CTP is a non-prescriptive process model. It describes the important software processes and what should happen in them, but it doesn’t put them in any order of improvement. This makes CTP a very flexible model. It allows you to identify and deal with specific challenges to your test processes. It identifies various attributes of good processes, both quantitative and qualitative. It allows you to use business value and organizational pain to select the order and importance of improvements. It is also adaptable to all software development lifecycle models.

Continue reading →

Analytical risk based testing offers a number of benefits to test teams and organizations that use this strategy. One of those benefits is the opportunity to make risk-aware release decisions.  However, this benefit requires risk based test results reporting, which many organizations have found particularly challenging.  This article describes the basics of risk based testing results reporting, then shows how Rex Black (of RBCS) and Nagata Atsushi (of Sony) developed and implemented new and ground-breaking ways to report test results based on risk.

Testing can be thought of as (one) way to reduce the risks to system quality prior to release. Quality risks typically include possible situations like slow system response to use input, incorrect calculations, corruption of customer data, and difficulty in understanding system interfaces. All testing strategies, competently executed, will reduce quality risks.  However, analytical risk based testing, a strategy that allocates testing effort and sequences test execution based on risk, minimizes the level of residual quality risk for any given amount of testing effort.

There are various techniques for risk based testing, including highly formal techniques like Failure Mode and Effect Analysis (FMEA). Most organizations find this technique too difficult to implement, so RBCS typically recommends and helps clients to implement a technique called Pragmatic Risk Analysis and Management (PRAM).  You can find a case study of PRAM implementation at another large company, CA here. While this article describes the implementation of the technique for projects following a sequential lifecycle, a similar approach has been implemented by organizations using Agile and iterative lifecycle models.

Continue reading →

Analytical risk based testing offers a number of benefits to test teams and organizations that use this strategy.  One of those benefits is the opportunity to make risk-aware release decisions.  However, this benefit requires risk based test results reporting, which many organizations have found particularly challenging.  This article describes the basics of risk based testing results reporting, then shows how Rex Black (of RBCS) and Nagata Atsushi (of Sony) developed and implemented new and ground-breaking ways to report test results based on risk.

Testing can be thought of as (one) way to reduce the risks to system quality prior to release.  Quality risks typically include possible situations like slow system response to use input, incorrect calculations, corruption of customer data, and difficulty in understanding system interfaces.  All testing strategies, competently executed, will reduce quality risks.  However, analytical risk based testing, a strategy that allocates testing effort and sequences test execution based on risk, minimizes the level of residual quality risk for any given amount of testing effort.

There are various techniques for risk based testing, including highly formal techniques like Failure Mode and Effect Analysis (FMEA).  Most organizations find this technique too difficult to implement, so RBCS typically recommends ­and helps clients to implement­ a technique called Pragmatic Risk Analysis and Management (PRAM).  You can find a case study of PRAM implementation at another large company, CA, here. While this article describes the implementation of the technique for projects following a sequential lifecycle, a similar approach has been implemented by organizations using Agile and iterative lifecycle models.

This article was originally published in Software Test and Quality Assurance www.softwaretestpro.com in their December 2010 edition.

Continue reading →

Organizing Manual Testing on a Budget

By Capers Jones, Vice President and Chief Technology Officer - Namcook Analytics LLC

RBCS is pleased to feature a special guest author for our newsletter article, Capers Jones. Capers Jones is, of course, a long-standing force for improving the software engineering industry, and has published a number of books that I consider essential reading for software professionals who seek to truly understand, through data and facts, what happens on software projects. Recently, he published an important book on software quality, The Economics of Software Quality. So, I asked Capers if he'd be willing to contribute a guest article, and he graciously agreed. This article, on software quality today and tomorrow, gives us a sobering view of our current situation, but also provides clear direction on what we need to do to get better. The good news is that we already have many of the tools we need to improve software quality. -- Rex Black 

In 2012 large software projects are hazardous business undertakings. More than half of software projects larger than 10,000 function points (about 1,000,000 lines of code) are either cancelled or run late by more than a year. 

When examining troubled software projects, it always happens that the main reason for delay or termination is due to excessive volumes of serious defects. Conversely, large software projects that are successful are always characterized by excellence in both defect prevention and defect removal. It can be concluded that achieving state of the art levels of software quality control is the most important single objective of software process improvements.

Quality control is on the critical path for advancing software engineering from its current status as a skilled craft to become a true profession.

Continue reading →

This article is excerpted from Chapter 3 of Rex Black’s book Managing the Testing Process, 3e.

A number of RBCS clients find that obtaining good test data poses many challenges. For any large-scale system, testers usually cannot create sufficient and sufficiently diverse test data by hand; i.e., one record at a time. While data-generation tools exist and can create almost unlimited amounts of data, the data so generated often do not exhibit the same diversity and distribution of values as production data. For these reasons, many of our clients consider production data ideal for testing, particularly for systems where large sets of records have accumulated over years of use with various revisions of the systems currently in use, and systems previously in use. 

However, to use production data, we must preserve privacy. Production data often contains personal data about individuals which must be handled securely. However, requiring secure data handling during testing activities imposes undesirable inefficiencies and constraints. Therefore, many organizations want to anonymize (scramble) the production data prior to using it for testing.

This anonymization process leads to the next set of challenges, though. The anonymization process must occur securely, in the sense that it is not reversible should the data fall into the wrong hands. For example, simply substituting the next digit or the next letter in sequence would be obvious to anyone­ it doesn’t take long to deduce that "Kpio Cspxo" is actually "John Brown" ­which makes the de-anonymization process trivial. 

Continue reading →

Defining Test Mission, Policy, and Metrics of Success

By Rex Black and Debbie Friedenberg

The International Software Testing Qualifications Board (ISTQB) program defines testing expansively. Static testing (via reviews and static analysis) is included as well as all levels of dynamic testing, from unit testing through to the various forms of acceptance testing. The test process is defined to include all necessary activities for these types of testing; planning; analysis, design, and implementation; execution, monitoring, and results reporting; and closure.

Continue reading →

An article featuring an interview with Rex Black was recently published in “Tester’s Life” magazine.   Click below to read the interview in English or, to see the original source article in it's entirety in Russian, visit www.testers-life.ru.  

Continue reading →

When I do assessments for clients, I talk to a lot of people, both inside and outside the testing group. In the opening moments of each interview, I try to engage in a friendly exchange, where I break the ice between the interviewee and myself. Not only is it more pleasant to have a friendly conversation than a tense one, but people are more open and honest with someone with whom they have some kind of positive relationship, compared to a complete stranger—or someone they see as hostile, cold, or inscrutable. Most of the time, I succeed, and I get to spend an interesting hour or so with someone who gives me the benefit of their insights and opinions.

The same is true, on a much larger and longer scale, for test managers. Testing is a matter of providing useful services to stakeholders. If those stakeholders have a good relationship with you and the other test managers in your test group, information will flow more smoothly in both directions. The job of the test group will become easier because it has better access to information it needs. The test group will also become more valuable because the information the group produces will flow more smoothly to the recipients of that information. It’s just human nature: We listen to and value the communications we receive from people we are comfortable with, and we are happy to reciprocate that flow of information.

It’s not that you must be a personal friend to every stakeholder with whom you work, but a good professional relationship with those stakeholders is a major factor in the success of a test manager. How well you and the other managers in the test group initiate, cultivate, and sustain these relationships will strongly influence the flow of information, as well as the support, you obtain from your colleagues.

A relationship is necessarily a two-way affair. You and the test group can’t be the only beneficiaries from a relationship, at least not a good one. Once, a person with whom I worked on a project described the CEO of one vendor as follows: “Every time I meet with that guy, I want to take a shower afterward,” meaning that he felt soiled just by being in the same room. Later in the project, when my colleague legitimately but accidentally came into possession of a memo that was certainly not in the vendor’s interests to disclose to its client, my colleague felt no compunction about copying the document before returning it in a way that did not disclose that he had seen it. The relationship had become two-way, but not in a good way.
As a contrast, I had an excellent relationship with this same vendor’s test manager. Across a significant cultural difference—the same difference my colleague and the CEO had not bridged—he and I forged a relationship of honesty and trust. I felt I could tell him the truth about what was happening on my side of the project, and he felt the same. We shared information to advance our mutual goals of a successful project and high-quality deliverable while at the same time respecting the limits on communication imposed by our different positions in terms of who our employers were. Even when the relationship between the two companies became testy, he and I were always able to communicate as friends with a good relationship of mutual respect.

I note that this anecdote does not represent an isolated incident but rather a truth that has become plain to me throughout my career in testing. The successful test manager, perhaps more than any other managers in the software business, must cultivate strong relationships with stakeholders, continuously reinforce those relationships with mutual benefits, and maintain the relationships through good times and bad. In the next few subsections, let’s look more closely at how.

Continue reading →

Back in the early days of computing, there was no such specific, separately identifiable activity known as “testing.” Developers debugged their code, and that was usually intertwined with some unit testing task. That didn’t work. My first job was as a programmer, and where I worked, this was exactly how we approached quality assurance. It was a quality disaster.

This approach by itself still doesn’t work. It does not work for those throwback, Neanderthal organizations that rely on this approach entirely. It does not work for those cutting-edge companies that think some fancy language or process or tool has solved the software quality problem at last. It does not work for anyone, unless we define “work” as “shift most costs of poor quality onto end users who are too stupid to know better or trapped in a monopolistic market without any real choice.” And those organizations that rely on stupid users or a monopoly position had better hope they are right.

With the widespread advent of independent test teams in the late 1980s and early 1990s, we saw improvements. I was working in an independent test lab in the late 1980s and as a test manager in the early 1990s, and we made great strides. However, we also saw the emergence of a new dysfunction, the “hurl it over the wall to the test guys and hold them responsible for delivered quality” mistake. Every now and then, we work with organizations that still suffer from this problem.

Let’s be clear. When quality matters—and shouldn’t it always—everyone must play a role. Good testing involves a series of quality assurance filters. Each team in the organization typically participates in and owns one or more of these filters.

Continue reading →

Bug Reporting Process

By Rex Black

Last time, I talked about an internal test process, managing test execution. This is a process that is only indirectly visible to the rest of the development team, in that, as long as you get through all your planned tests, effectively respond to change, and report your findings intelligibly, then for all most people care your testing could occur through voodoo, augury, and a Ouija board.

Continue reading →

Why Care About Testing Processes?

  • Bugs are expensive
    US economy loses $60B yearly due to bugs, $20B of which could be saved through better testing
  • Bugs drive away customers
    Handwriting recognition bugs in Newton cost Apple the person digital assistant (PDA) market
  • Bugs sometimes kill people
    Therac-25 overdoses, Patriot clock overflow
  • Inadequate testing endangers the project
    Bad testing processes can increase the risk of project delay or cancellation by 25 to 300% 

Continue reading →

As a test professional with almost two decades in the computer field, I’m pleased to see a new publication devoted to our field, the Journal of Software Testing Professionals, and honored to serve as a Contributing Editor. As my first task in this role, I have been asked to write and collaborate with other software testing professionals to produce a series of articles addressing what we consider critical processes that the test professional must master in order for her to succeed. In each of these articles, we’ll examine one critical process closely, offering practical advice on what works and caveats on what can cause problems

Continue reading →

So, you are responsible for managing a computer hardware or software test project? Congratulations! Maybe you’ve just moved up from test engineering or moved over from another part of the development team, or maybe you’ve been doing test projects for a while. Whether you are a test manager, a development manager, a technical or project leader, or an individual contributor with some level of responsibility for your company’s test and quality assurance program, you’re probably looking for some ideas on how to manage the unique beast that is a test project.

Continue reading →

There are two measures that have a strong influence on the outcomes of software projects: 1) Defect potentials; 2) Defect removal efficiency.

The term “defect potentials” refers to the total quantity of bugs or defects that will be found in five software artifacts: requirements, design, code, documents, and “bad fixes” or secondary defects. 

Continue reading →

Mission Made Possible

By Rex Black and Greg Kubaczkowski

Your mission should you choose to accept it: A client needs to implement an integrated, automated unit, component, and integration testing process. The system development teams are spread across 3 continents. They use multiple platforms. Their system architecture is interdependent and complex. Some tests need to be run on an application server. Both static and dynamic testing is required. The process you design must be simple to use, not an additional burden for busy developers. It should be automated and easily integrated. By the way, only 3 designers have been assigned to this mission. You have 6 weeks. Good luck. This message will self-destruct in 20 seconds… 

Continue reading →

In the last issue, I dealt with a collaborative test process, getting test releases into the test lab. In this article, let’s look instead at an internal test process, managing the execution of tests against a test release. One can think of the test manager’s role during the test execution process as one of managing a search for the unexpected. While all managers are expected to manage unexpected events within their areas of responsibility, the test manager is the only manager in the software development organization whose area of responsibility is searching for the unexpected. This search takes the form of running a set of tests. How does a test team take these tests, run them against an installed system, and capture information about the system under test, the test cases, and the test execution process itself? 

Continue reading →

Project managers must develop quality systems that provide the needed features in a timely and cost-effective fashion. Testing systems prior to release is necessary to assess their quality, but what’s the return on that testing investment? This article describes four quantifiable metrics for testing ROI

Continue reading →

Recently, we worked on a high-risk, high-visibility system where performance testing (“Let’s just make sure it handles the load”) was the last item on the agenda. As luck would have it, the system didn’t handle the load, and very long days and nights ensued. Delivery was late, several serious disasters were narrowly averted, and large costs were incurred.

Continue reading →

Most software test teams exist to assess the software’s readiness prior to release. To achieve this goal, two primary tactics are used:

  1. Execute test cases or scenarios that are likely to find errors, resemble actual usage, or both.
  2. Report the test results, the defects found, and defects fixed, which, collectively, make up the test status and reflect the quality of the software.

For the test manager, the first task category primarily involves managing inward: assembling the test team and test resources, designing a solid test system, implementing the test system, and running the tests in an intelligent, logical fashion. The second area of responsibility involves upward and outward management. Upward management includes how you communicate with your managers, other senior managers, and executive staff. Outward management includes communication with management peers in development, configuration/release management, sales, marketing, finance, operations, technical support, and so on. As a test manager, your effectiveness in reporting test status has a lot to do with both your real and perceived effectiveness in your position. To put it another way, your management superiors and peers measure your managerial competence as much by how well you report your results as by how you obtain your results. 

Continue reading →

Realizing a solid return on your testing investment requires smart selection of tests. Cost of quality analysis tells us that it’s cheaper to find and fix bugs before the customers do, but, to keep bugs away from customers, we have to find the ones that matter. Doing so requires that we understand how the customers will use the system. 

Continue reading →

Automated test tools are powerful aids to improving the return on the testing investment when used wisely. Some tests inherently require an automated approach to be effective, but others must be manual. In addition, automated testing projects that fail are expensive and politically dangerous. How can we recognize whether to automate a test or run it manually, and how much money should we spend on a test? 

Continue reading →

Pervasive testing means getting the right people working together through the right processes at the right time for high-ROI testing. Pervasive testing is the final piece of the test investment puzzle, and just as important as all the others. Through pervasive testing, all the ideas we’ve explored so far come together.

Continue reading →

Subtle but catastrophic bugs, such as those that cause server crashes and database record-lock race conditions, often only reveal themselves during performance, stress, volume, data quality, and reliability testing. Such testing is most effectively performed in test environments— hardware, software, network, and release configurations—that mimic as nearly as possible the field environment, because test results in less-complex settings often do not extrapolate due to the non-linearity of software. In complex settings, such as Web and e-commerce server and database farms, managing these lab configurations can be quite challenging. This paper presents a basic Access database, designed using the Entity-Relationship technique, that will allow the Test Manager to plan, configure, and maintain this test environment through the test project. 

Continue reading →

Information appliances, which provide simplified, easy access to specific information such as e-mail and Web sites, promise to bring the benefits of computing to a wide customer base, including some computer-averse people who have hitherto avoided buying a computer. Internet appliances are evolving from personal computers, game stations, digital mobile phones, and server technologies. While this allows us to apply well-known quality assurance techniques, including testing techniques, the software quality professional must remember that the risks to product quality are different; the quality bar is higher, especially in terms of usability, robustness, and harmonizing the appliance with the dynamic Internet. Customers will assess the quality of information appliances by the degree to which the appliance reliably, quickly, transparently, and intuitively provides them with access to the desired information, and we expect them to be much less understanding of glitches than the current PC user. Information appliances are gaining wide acceptance—millions will hit the market in the next few years—so many of us who practice software quality professions will spend time working on projects to develop them. Indeed, we expect that information appliances will present tremendous opportunities to those who seek to bring quality to software in the new millennium. This paper presents the test team’s findings on one such project.

Continue reading →

In the Test Manager’s perfect world, her team would be staffed entirely by expert software engineering professionals. These test engineers could converse intelligently one minute with development engineers about code coverage and memory leaks; the next minute talk with technical support agents about the customer’s experience of quality; and one minute later speak to marketeers about trade-offs between features, schedule, budget, and quality in the upcoming release. Such engineers would spend their time programming unit and integration test stubs and scaffolds for structural testing, scripting automated behavioral tests at the GUI level, and creating load generators and performance probes. Manual testing would be used sparingly to fill the gaps in such tests, for example by producing interesting error conditions and checking the system’s response. The Test Manager would have ample staffing budget to hire such peer-level test engineers, and would have early involvement in the development effort. He could focus on verifying quality risk coverage using techniques like failure mode and effect analysis, analyzing and reporting defect management metrics, and setting up career development plans for his engineers.

Continue reading →

Suppose you went to a restaurant for dinner, sat down, and told the waiter, "Bring me dinner and a drink." You didn't provide any further details, though you had something specific in mind. What are the chances that you'll get the dinner and drink you expected? While no one would ever do this in a restaurant, it happens sometimes on projects that involve third parties.

If we have certain expectations and requirements for an engagement with a third party, those should be defined and clearly communicated between the parties. The best practice is to have that definition and communication before the project starts and to put the agreed-upon terms into the contract. If the third party is delivering software, then these requirements should include quality targets, including measurements of those targets. The measurements should be objective and not subject to distortions.

In addition to defining the requirements, the point at which those requirements must be met should be defined. This can be done by defining entry and exit criteria that establish quality gates for deliverables. Because these quality gates will control the start and end of project phases, they should be synchronized with the phases of the project and aligned with project schedule milestones. 

Continue reading →

Project Retrospective

By Rex Black

An excerpt from The Expert Test Manager: Guide to the ISTQB Expert Level Certification book by Rex Black, Jim Rommens and Leo Van Der Aalst due to be published by Rocky Nook. All material is provisional and may be subject to change
 
As a colleague told me once, a good motto for software teams is: "Make interesting new mistakes." His explanation was, since you are human, you'll make mistakes. But you should make interesting ones, ones you can learn from, and you should only make a mistake once.  How do you ensure that you make only interesting new mistakes? By learning from each mistake that you make. How to you learn from each mistake? In this short article, you'll read about a proven technique for learning from mistakes, retrospectives. Useful in both Agile and traditional lifecycles, this simple technique can make you and your colleagues a process improvement machine!
 

Continue reading →

"Back in 2010, at the launch of Core Magazine, http://www.coremag.eu/, I wrote a series of columns to welcome people to the magazine. As a sort of Throw-Back-December, here they are, as they appeared in the original magazine issues. I hope you enjoy them."
-Rex Black
 
Greetings, and welcome to my quarterly column on software testing best practices.  When I was asked to write this column, I had to choose the approach, the theme.  The writers' aphorism says, "Write what you know." So, what do I know?
 
Well, if you know me and my consulting company, RBCS, you know that we spend time with clients around the world, in every possible industry, helping people improve their testing with training or consulting services, or doing testing for them with our outsourcing services.  Our work gives me insights into what goes on, the actual day-to-day practice of software testing.
 

Continue reading →

Seven Steps to Reducing Software Security Risks

By Rex Black, President, RBCS, Inc.

If you are a software tester, programmer, or manager, you probably know that developing secure software is no longer simply desirable—it’s completely essential.  

Some might assume that most security problems arise from the operating system or networking layers, well below the application code they are working on.  However, figures for Web-based applications show that over three-quarters of security exploits arose from applications. 

So, you know you need secure code, but how to get there? What are your security risks?  What security failures and bugs do you have?  What do these security risks, failures, and bugs mean?  How can you reduce security risk in a way that doesn’t create new problems?  How do you monitor my progress over time?  This article will outline seven steps that will allow you to answer these and other questions as you improve your software’s security.

Continue reading →

This is an excerpt from my book, Expert Test Manager, written with James Rommens and Leo van der Aalst. I hope it helps you think more clearly about the test strategies you use.

A test policy contains the mission and objectives of testing along with metrics and goals associated with the effectiveness, efficiency, and satisfaction with which we achieve those objectives. In short, the policy defines why we test. While it might also include some high-level description of the fundamental test process, in general the test policy does not talk about how we test.

The document that describes how we test is the test strategy. In the test strategy, the test group explains how the test policy will be implemented. This document should be a general description that spans multiple projects. While the test strategy can describe how testing is done for all projects, organizations might choose to have separate documents for various types of projects. For example, an organization might have a sequential lifecycle test strategy, an Agile test strategy, and a maintenance test strategy.

Continue reading →

Test-Driven Development, Acceptance Test-Driven Development, and Behaviour-Driven Development

By by Rex Black, Marie Walsh, Gerry Coleman, Bertrand Cornanguer, Istvan Forgacs, Kari Kakkonen, and Jan Sabak

[Note: This is an excerpt from Agile Testing Foundations: An ISTQB Foundation Level Agile Tester Guide, by Rex Black, Marie Walsh, Gerry Coleman, Bertrand Cornanguer, Istvan Forgacs, Kari Kakkonen, and Jan Sabak, published July 2017. Kari Kakkonen wrote this selection. The authors are all members of the ISTQB Working Group that wrote the ISTQB Agile Tester Foundation syllabus.]

The traditional way of developing code is to write the code first, and then test it. Some of the major challenges of this approach are that testing is generally conducted late in the process and it is difficult to achieve adequate test coverage. Test-first practices can help solve these challenges. In this environment, tests are designed first, in a collaboration between business stakeholders, testers, and developers.  Their knowledge of what will be tested helps developers write code that fulfils the tests. A test-first approach allows the team to focus on and clarify the expressed needs through a discussion of how to test the resulting code. Developers can use these tests to guide their development.  Developers, testers, and business stakeholders can use these tests to verify the code once it is developed.

A number of test-first practices have been created for Agile projects, as mentioned in section 2.1 of this book. They tend to be called X Driven Development, where X stands for the driving force for the development. In Test Driven Development (TDD), the driving force is testing. In Acceptance Test-Driven Development (ATDD), it is the acceptance tests that will verify the implemented user story. In Behaviour-Driven Development (BDD), it is the behaviour of the software that the user will experience. Common to all these approaches is that the tests are written before the code is developed, i.e., they are test-first approaches. The approaches are usually better known by their acronyms. This subsection describes these test-first approaches and information on how to apply them is contained in section 3.3.

Test-Driven Development was the first of these approaches to appear. It was introduced as one of the practices within Extreme Programming (XP) back in 1990. It has been practiced for two decades and has been adopted by many software developers, in Agile and traditional projects. However, it is also a good example of an Agile practice that is not used in all projects. One limitation with TDD is that if the developer misunderstands what the software is to do, the unit tests will also include the same misunderstandings, giving passing results even though the software is not working properly. There is some controversy over whether TDD delivers the benefits it promises. Some, such as Jim Coplien, even suggest that unit testing is mostly waste.

TDD is mostly for unit testing by developers.  Agile teams soon came up with the question: What if we could have a way to get the benefits of test-first development for acceptance tests and higher level testing in general? And thus Acceptance Test-Driven Development was born.  (There are also other names for similar higher-level test-first methods; for example, Specification by Example (SBE) from Gojko Adzic.) Later, Dan North wanted to emphasize the behaviours from a business perspective, leading him to give his technique the name Behaviour-Driven Development. ATDD and BDD are in practice very similar concepts. 

Let’s look at these three test-first techniques, TDD, ATDD, and BDD, more closely in the following subsections.

 

Continue reading →

Webinars

In Praise of Specialization

Recorded August 27, 2014

Towards a True Engineering Profession

Recorded February 18, 2014

The Sherpas of Testing Success

Recorded August 14, 2013

Profiles of Failure

Recorded April 24, 2013

Hiring Great Testers

Recorded November 10, 2010

Test Organization Options

Recorded July 1, 2010

Satisfying Test Stakeholders

Recorded May 20, 2010

Test Reporting for Impact

Recorded November 23, 2009

The Future of Test Management

Recorded October 8, 2009

RBCS Videos: RBCS Services

Recorded August 18, 2009

Ten Worst Things for Testing

Recorded June 9, 2009

Agile Testing Challenges

Recorded April 30, 2009

Podcast Episodes

Software testing challenges with Agile 11/16/2009


Length: 1h 30m 21s

Is your company going Agile? What does that mean for testing? What should you expect? Are you curious? Are you excited? A number of companies have adopted Agile methodologies, for various reasons ranging from speed of delivery, flexibility in rapidly changing situations, or just plain trendiness. Every software development lifecycle model, sequential or spiral, incremental or Agile, has testing implications. Some of these implications ease the testing process. Some of these testing implications challenge testing. In this Webinar, Rex Black discusses the key Agile testing challenges, so that you can recognize and resolve these challenges if your company adopts an Agile approach.

Listen now →

Ten Worst Things for Testing 12/14/2009


Length: 1h 29m 44s

Lots of people give top 10 lists, but only the very brave will give a bottom 10 list! Let's face it though: Smart people sometimes do dumb things. Smart testers and test managers are no exception. In this talk, Rex Black will discuss things he's done and things he's seen other smart test professionals do that, upon close reflection, turn out to be big mistakes. That way, you can go back to work the next day and not do them.

Listen now →

Six Surprises from Test Assessments 12/28/2009


Length: 1h 22m 10s

RBCS consultants, led by Rex Black, travel the globe performing services for various clients, including assessments of test teams, test processes, and organizational quality outcomes. Often, our assessment findings surprise the assessed organization-and sometimes they even surprise us! In this eye-opening Webinar, filled with case studies backed by data, not just opinion, Rex Black will sample six surprising test assessment findings that span the critical testing processes, from organizations around the world.

Listen now →

The Future of Test Management 1/27/2010


Length: 1h 30m 22s

Outsourcing. Open source. Legacy systems. Too much testing in too little time with too few resources. Simultaneous skills gluts and skills shortages. Choosing the right certifications for testers. Systems of systems. Iterative and Agile methodologies. Connecting to business value. These are ten systemic, long-term topics that confront the current and future test managers. The successful test manager will enter the future ready to handle these issues, while other test managers will become former test managers due to their failure to handle them. In this webinar, Rex Black will talk about these challenges and possible solutions to them. You will leave better able to survive and thrive in the coming twenty-teen decade.

Listen now →

Test Reporting for Impact 02/08/2010


Length: 1h 26m 57s

Testing is about producing information, and effective testing is about communicating that information to the various test stakeholders in a way that makes an impact on their thinking, their understanding, and their choices for how the project moves forward. Testers, test leads, and test managers frequently have to report on test status and on test analysis, answering important questions about bugs, progress, coverage, the meaning of the results, and the causes of the observed outcomes. In this talk, Rex Black will give examples of test status and analysis graphs and charts that have helped him achieve test reporting for impact.

Listen now →

Performance, load, and reliability tests are hard to design, hard to run, and essential in today’s complex IT environments. You need to apply lots of smarts, the right tools, and a pinch of rigor to make it work. In this Webinar, Rex Black will explain important performance, load, and reliability tests such as environments, load generation, models and simulation, tools, and timing. This Webinar will be illustrated by real-world examples drawing on Rex’s twenty years of experience with performance, load, and reliability testing.

Listen now →

What is it about testing, done really well, that provides long-term satisfaction to the stakeholders? Who are the stakeholders and what do they want from testing? What are the external and internal forms of value delivery in testing that truly satisfy? How can testers and test managers build testing organizations that provide this satisfaction over the long term? How can we, as test professionals, create elegance, effectiveness, efficiency, and even delight for ourselves and our stakeholders in our work? Join this webinar to learn how to succeed as a test manager and a test team by delivering value that satisfies your stakeholders.

Listen now →

Software Testing: Test Organization Options 9/6/2010


Length: 1h 26m 14s

How should we staff and organize our test teams? This is a great challenge for many test managers, and indeed the organizations that have test teams. Should we have specialized or generalized test teams? What role (if any) do people outside the testing team play in testing? Should we have an in-house testing center of excellence or testing services organization, and, if so, what should that team do? When and how should we use outsource testing service providers? How do lifecycle options like the V-model, iterative, agile, and spiral affect the answers to these questions? These are hard questions that many RBCS clients struggle with. Join this webinar to review and discuss the options available, in an interactive forum, with Rex Black. Attendees are encouraged to send questions and topics to cover to Rex prior to the webinar.

Listen now →

Software Testing: Hiring Great Testers 2/1/2011


Length: 1h 36m 19s

As the worldwide economy recovers, many test managers are facing a happy challenge: How do I hire really great testers? Given that a great employee can be ten times as effective as a poor employee, hiring the right people is a critical question for us as managers. In this webinar, Rex will help you find answers to this question. We’ll examine what attributes you should look for in great testers. We’ll talk about skills testers need, and how to measure and grow those skills. We’ll talk about how to interview testers to make smart hiring decisions. Join this webinar, illustrated with examples throughout, to learn ways to hire and grow a great test team.

Listen now →

While much of testing and test management involves rational decision-making, measuring quality and providing testing services to the team, there are realms of the test manager’s job where psychology meets politics to form psychopolitics. In this webinar, drawn from three decades of industry experience and materials in his best-selling book Managing the Testing Process, 3rd edition, Rex will discuss how psychology and politics can collide to make the test manager’s job…interesting. Join in the discussion after the initial presentation with your own questions and stories about testing psychology, testing politics, and plain ol’ psycho-politics! The International Software Testing Qualification Board has already effected profound change in the software testing profession, and the rate of that change continues to accelerate. Not only have 150,000 people have achieved Foundation certification, but increasing numbers of testers are taking Advanced exams, including the extremely challenging Advanced Technical Test Analyst exam. In this webinar, Rex will explain why it’s so tough—and why technically-oriented testers who have achieved Foundation certification should step up to the challenge.

Listen now →

Five years ago, Rex gave a talk in Tokyo, Wellington, Sydney, and Portland where he talked about five trends which were affecting testing, putting testers’ careers at stake, creating both risks and opportunities. The trends: globalization; test automation; commoditization; compliance, regulation, and tort law; and, education and certification. In this webinar, Rex will revisit each of those five trends now, five years on. He’ll talk about the effects that have occurred, the effects that have not occurred, and the effects that are yet to occur. He’ll give you ideas about how to effectively handle these trends, and also some hints about new trends that are emerging.

Listen now →

How can independent test teams guide an organization to success? What organizational options exist for such teams? How does independent testing fit into the software lifecycle? How can we measure success for such team? In this presentation, drawn from three decades of industry experience, Rex will help you find answers to these questions, and to establish—or improve—your independent testing team. Join in the discussion after the initial presentation, and ask Rex about the challenges you face with test estimation.

Listen now →

In a perfect world, we would have boundless time and resources to deliver systems with plentiful features, each feature having perfect quality. In the real world, we must balance schedule, budget, features, and quality. When schedule, budget, and features get all of the attention, we have a quality quandary. In this webinar, Rex will discuss ways that test professionals can help their project teams resolve that quandary. First, it’s important to recognize the trade-offs being made, and then test professionals can promote the five elements of making successful trade-offs: shared vision; disciplined management; quality in, and bugs out, throughout; focused testing; and, sending the right message. Rex will illustrate these five elements with a variety of case studies and examples.

Listen now →

In 2003, Rex’s book, Critical Testing Processes, came out. In it, Rex introduced a test process framework based on criticality, and focused on business value. With the 2012 publication of Rex’s new e-book, Improving the Testing Process, Rex provides a clear set of guidelines on how to use the Critical Testing Processes framework to assess and improve your testing processes. It includes questions to ask and metrics to evaluate. In this webinar, Rex will explain the evaluation guidelines and how to apply them to your organization. You’ll come away ready to get started on bottom-line focused, data-driven improvements to your testing processes.

Listen now →

Some testing teams see themselves as quality cops or process cops, there to enforce entry criteria, methodology standards, work product delivery dates, and other policies on developers, business analysts, sales and marketing staff, and other project team members. While this can be valuable and appropriate in some organizations, most of the time it is not a winning position for test teams. What we see with our clients is that test teams who deliver the most value are those that focus on being a trusted partner to their stakeholders, helping them deliver the highest quality software and systems on time and within budget. In this webinar, illustrated with case studies of successful and unsuccessful test teams, Rex will explain how you can transform your test team into a trusted partner.

Listen now →

Are test management and quality management the same? Are they different? How are they related? Many people are confused about these questions, and that confusion results in many problems for test teams around the world. Rex Black has seen that confusion first-hand, and has helped clients resolve the problems that confusion can cause. In this talk, Rex will provide five key lessons in how to recognize the confusion and resolve problems, illustrated with case studies throughout. You will leave this session ready to provide clear leadership in test management and quality management, and to help your organization achieve successful testing and quality.

Listen now →

This is another webinar in a series on advanced software testing, excerpted from Rex’s popular books, Managing the Testing Process, third edition, and Advanced Software Testing: Volume II, a book for test managers. In this webinar, Rex will discuss test policies, strategies and plans. We’ll cover various topics that test managers must consider to make sure that test goes as smoothly as possible. What should testing accomplish and how do we recognize success? How should we approach testing? What specific activities need to be planned for each project? What risks can affect testing, and how can we manage those risks in our test plans? How does lifecycle affect these documents? Join this webinar, illustrated with examples throughout, to learn techniques that you can apply to your work right away.

Listen now →

Profiles in Failure: 4/24/13


Length: 1h 27m 23s

You can often learn a lot more from a failed project than you can from one that succeeds. So, in this free webinar, Rex offers detailed case studies of three projects that failed—even though they could have succeeded. Each project was very different, but common themes emerge: earlier testing, better quality throughout the project, tighter collaboration between testers and other stakeholders, and better vendor quality management. Rex will discuss how to you can learn the lessons that could have saved these projects so you can avoid being a profile in failure on your next project.

Listen now →

Webinar: The Seven Deadly Sins of Testing: 6/11/13


Length: 0h 53m 36s

Many smart, otherwise-capable testers sabotage their own careers by committing one or more of the deadly sins of testing. 1. Irrelevance/redundancy 2. Ignorance of relevant skills or facts 3. Obstructionism 4. Adversarialism 5. Nit-picking 6. Blindness to project/organizational priorities 7. Last-moment-ism Are you your own worst enemy? Come join us to discuss these seven deadly sins with Rex. You might recognize your own behaviors, or behaviors of others on your test team. Rex gives examples of these behaviors through case studies, and tells you how to stop the behaviors and solve the problems those behaviors have created. For sinners and non-sinners alike, Rex offers ideas on how to become a testing saint.

Listen now →

Webinar: The Sherpas of Testing Success 08/14/13


Length: 1h 2m 26s

How can independent test teams guide an organization to success? What organizational options exist for such teams? How does independent testing fit into the software lifecycle? How can we measure success for such team? In this presentation, drawn from three decades of industry experience, Rex will help you find answers to these questions, and to establish—or improve—your independent testing team. Join in the discussion after the initial presentation, and ask Rex about the challenges you face with test estimation.

Listen now →

Webinar: Towards a True Engineering Profession: 2/18/14


Length: 1h 14m 28s

The software profession is evolving rapidly, and software testing is also evolving. Some people talk about software and software testing as a “craft,” but are we doomed to remain in such a primitive state, like medieval swordsmith’s guilds? Or, as the phrases “software engineering” and “test engineer” have long implied, can we advance to a true engineering profession. What is the history? What can other engineering professions teach us? What do other engineering professions have, and how can we go about creating analogous resources for software engineering? What will a true software engineering profession mean for testing? Join Rex for this imaginative trip into one possible future, based on science and engineering facts.

Listen now →

Webinar: Interviewing Testers: 5/30/14


Length: 1h 38m 24s

Did you know that dealing with a single problem employee can consume as much as 50% of his or her manager’s time? Imagine how that reduces the manager’s effectiveness! This is why making smart hiring decisions is one of the most important and central roles for test leads and test managers. However, it’s also one of the hardest roles we face. In this webinar, Rex Black, having hired dozens of testers and coached managers around the world on how to improve their hiring practices, will share some of the techniques you can use to ensure that your next hiring decision is the best one you’ve made—so far.

Listen now →

Webinar: In Praise of Specialization: 8/27/14


Length: 1h 32m 19s

The hot trend these days is to talk about cross-functional, self-organizing teams, and to disparage specialization. For example, Ken Schwaber wrote, “You don’t have to be a tester to test, or a designer to design.” But is that really true? Would you accept the following statements? If you’re a baseball player, you don’t have to be a pitcher to pitch, or a catcher to catch. If you’re a musician, you don’t have to be a violinist to play violin, or a pianist to play piano. If those statements make you uncomfortable, you’re not alone. In this webinar, Rex will explain why specialization matters. He will explain the science and psychology behind specialization. He will offer three specific case studies where a lack of specialized skills contributed to dramatic failures, and three case studies where the presence of skills averted similar failures. You’ll go away with some food for thought, and perhaps a determination to reorganize the way you manage your teams.

Listen now →

Webinar: Interviewing Testers: 1/2/15


Length: 1h 26m 38s

This is an encore presentation of one of our most popular webinars. If you missed it the first time, you can catch it now. Did you know that dealing with a single problem employee can consume as much as 50% of his or her manager’s time? Imagine how that reduces the manager’s effectiveness! This is why making smart hiring decisions is one of the most important and central roles for test leads and test managers. However, it’s also one of the hardest roles we face. In this webinar, Rex Black, having hired dozens of testers and coached managers around the world on how to improve their hiring practices, will share some of the techniques you can use to ensure that your next hiring decision is the best one you’ve made—so far.

Listen now →

What would John Connor, son of Sarah Connor, hero of the resistance in the “Terminator” movie series, say about the Internet of things, the ubiquity of mobile devices, the fact that we almost ran out of IP addresses, software that updates itself, and other signs of the coming computer apocalypse? He’d probably tell us to pull the plug, quickly! Beyond the nightmare scenario of the “Terminator” films, what are the implications of quality, and lack of quality, for the now-real situation that everything is connected to everything else? Will it be Neuromancer, “Terminator,” androids dreaming of electric sheep, or something more benign? Join us for some fun and interesting speculation and prediction on the future of limitless connectivity.

Listen now →

Webinar: Leading a Test Team: 3/26/15


Length: 1h 21m 40s

In many ways, leading a test team is like leading any other team. After all, people are people, no matter what work they do or where they live. Of course, there are some aspects that are specific to managing technical teams. And there are a few aspects that are specific to managing a test team. Given a good team, which you’ve developed carefully, how can you get the best work from them and keep them engaged and motivated? Rex will explore some ideas, and take your questions, in this free webinar.

Listen now →

Did you ever wonder why your colleagues sometimes look at you funny when you’re explaining your test results, or ask you strange questions like, “Why don’t you find all the bugs in testing?” Maybe they don’t understand what you do or even why you do what you do. In this free webinar, Rex will reveal ten common misunderstandings that your fellow software professionals have about testing, and discuss ways to resolve those misunderstandings to promote better communication.

Listen now →

Webinar: Schools of Testing Debate 4/15/2014


Length: 1h 10m 9s

“Schools of Testing”: Useful Paradigm or Negative Influence? Are there really four different “schools of testing”? Do you have to belong to one? Or are there just different testing strategies, to be selected and blended as appropriate? Has the concept of “schools of testing” promoted clearer thinking about testing, or has this concept actually created a useless schism, sowing conflict and slowing growth of the profession? In this debate, Rex Black argued the negative side of the case, being one of the leading critics of the concept of “schools of testing.” One the other side, you’ll hear one of the originators of the concept, Cem Kaner, argue for its benefits. Love it or hate it, you can hear both sides and make up your own mind.

Listen now →

Webinar: Why Do You Test? 9/8/15


Length: 1h 22m 48s

Why do you test? Have you asked yourself that question before? Try it. Then take the next step: Identify two or three colleagues with different roles and ask them why they think you test. Typically, the lists won’t match up very well, and many times the lists include completely unrealistic objectives like, “Make sure the software works right before we put it into production,” or, “Find all the bugs before we ship.” Instead, in this webinar, Rex will explain a process that will allow you to create a set of clearly defined, achievable, measurable set of objectives. This process will also build consensus on those objectives within your team and across teams. Tune in to this free webinar to step out of the confusion and into clarity on this important topic.

Listen now →

Webinar: The Seven Deadly Sins of Testing


Length: 1h 5m 46s

Many smart, otherwise-capable testers sabotage their own careers by committing one or more of the seven deadly sins of testing.  Are you your own worst enemy?  Come join us to discuss these seven deadly sins with Rex.  You might recognize your own behaviors, or behaviors of others on your test team.  In this updated webinar presentation, Rex gives examples of these behaviors through case studies, and tells you how to stop the behaviors and solve the problems those behaviors have created.  For sinners and non-sinners alike, Rex offers ideas on how to become a testing saint.  

Listen now →

Webinar: Two Bug Metrics, Millions in Process Improvement


Length: 1h 20m 49s

When we do assessments, we always try to look at process metrics. In most cases, we can find millions of dollars in process improvement opportunities. In this webinar, Rex will show you how two very simple bug metrics, calculated using only two simple facts for each bug report using simple, free spreadsheets you can get from our website, can reveal millions and millions of dollars in potential process improvements. All the more reason to track those bugs! To paraphrase Timothy Leary: Tune in, download, and drop software co

Listen now →

Webinar: The Psychopolitics of Test Management 1/11/17


Length: 1h 21m 33s

While much of testing and test management involves rational decision-making, measuring quality and providing testing services to the team, there are realms of the test manager’s job where psychology meets politics to form psychopolitics.   In this webinar, drawn from three decades of industry experience and materials in his best-selling book Managing the Testing Process, 3rd edition, Rex will discuss how psychology and politics can collide to make the test manager’s job…interesting. Join in the discussion after the initial presentation with your own questions and stories about testing psychology, testing politics, and plain ol’ psycho-politics!

Listen now →

Stupid Metrics Tricks and How To Avoid Them


Length: 1h 1m 48s

If you have been in software engineering for a while—or in fact just in the working world in general for a while—you’ve probably seen someone do something stupid with metrics. Such mistakes raise a whole bunch of interesting questions. What are the most common metrics mistakes? Why are they mistakes? Why do people make these mistakes? Are you making these mistakes? Why use metrics at all, when there are so many mistakes? In this talk, Rex will give real-world examples of these mistakes, explain the management and economic theories behind metrics, and help you find ways to implement metrics that aren’t stupid.

Listen now →

Training

ISTQB Virtual Expert Test Management Strategic Test Manager Boot Camp

The Expert Test Management Strategic Test Manager Boot Camp, created by Rex Black, past President of the International Software Testing Qualifications Board (ISTQB), past President of the American Software Testing Qualifications Board (ASTQB) and co-author of the International Software Testing Qualifications Board Expert Level Syllabus Test Management, is ideal for testers and test teams preparing for certification in a short timeframe with time and money constraints. 

View details →

ISTQB Virtual Expert Test Management Operational Test Manager Boot Camp

The Expert Test Management Operational Test Manager Boot Camp, created by Rex Black, past President of the International Software Testing Qualifications Board (ISTQB), past President of the American Software Testing Qualifications Board (ASTQB) and co-author of the International Software Testing Qualifications Board Expert Level Syllabus Test Management, is ideal for testers and test teams preparing for certification in a short timeframe with time and money constraints. 

View details →

ISTQB Virtual Expert Test Management Managing the Test Team Boot Camp

The Expert Test Management Managing the Test Team Module Boot Camp, created by Rex Black, past President of the International Software Testing Qualifications Board (ISTQB), past President of the American Software Testing Qualifications Board (ASTQB) and co-author of the International Software Testing Qualifications Board Expert Level Syllabus Test Management, is ideal for testers and test teams preparing for certification in a short timeframe with time and money constraints. 

View details →

ISTQB Expert Test Management Strategic Test Manager Training

The Expert Test Management Strategic Test Manager course, created by Rex Black, past President of the International Software Testing Qualifications Board (ISTQB), past President of the American Software Testing Qualifications Board (ASTQB) and co-author of the International Software Testing Qualifications Board Expert Level Syllabus Test Management, is ideal for testers and test teams preparing for certification. 

This hands-on course provides test engineers with the ability to define and carry out the tasks required to put the strategy into action and is ideal for testers and test teams preparing for certification. In preparation for the exam, participants will learn the subject matter behind the test standard and deepen their understanding by working through case studies and exercises. In group exercises, typical review situations are played out and analyzed.

View details →

ISTQB Virtual Expert Test Management Strategic Test Manager Training

The Virtual Expert Test Management Strategic Test Manager course, created by Rex Black, past President of the International Software Testing Qualifications Board (ISTQB), past President of the American Software Testing Qualifications Board (ASTQB) and co-author of the International Software Testing Qualifications Board Expert Level Syllabus Test Management, is ideal for testers and test teams preparing for certification. 

This hands-on course provides test managers with the ability to define and carry out the tasks required to put the strategy into action and is ideal for testers and test teams preparing for certification. In preparation for the exam, participants will learn the subject matter behind the test standard and deepen their understanding by working through case studies and exercises. In group exercises, typical review situations are played out and analyzed.

View details →


Copyright ® 2017 Rex Black Consulting Services.
All Rights Reserved.
ISTQB Logo ASTQB Logo IREB Logo PMI Logo ISTQB Logo
PMI is a registered mark of the Project Management Institute, Inc.