RBCS COVID-19 response: All of our public training courses through May will be run virtually (view details).
The last one isn't actually a template, but it's something many people find interesting.
If there's some other template you need, search the Basic Library and Advanced Library. If you still don't find it, post a comment here on the blog letting the readers and me know what you're looking for. Maybe someone can help.
Remember, though, a template is not an excuse to turn your brain off. Be sure to use templates thoughtfully.
I took a few moments today to record another video blog entry, which you can find at Regional Software Testing Immaturity: Fact or Fallacy?
Here's the synopsis: I'm in Los Angeles, on my way to the STANZ conference in Australia and New Zealand. That and other recent international trips have gotten me thinking about something that I often hear at such international conferences, which are comments along the lines of, "You know, software testing as a profession and a practice is really immature in region X," where region X might or might not be where I'm at. Based on my experience with clients around the world, though, the gap isn't as big as people often think it is. Is software testing actually significantly less mature in some regions than others? What has your experience been? I'd be interested in opinions, case studies, and stories from you, especially the many international readers of this blog but also people in North America.
A couple recent events might seem to indicate a greater appetite--and need--for greater technical skills. I thought I'd leave a quick video blog post on this topic--see Technical Testers Blog. Let me know what you think. Are we in need of stronger technical skills? Do you have technical skills? Do you need and want technical skills?
When I wrote my book Critical Testing Processes in the early 2000s, I started with the premise that some test processes are critical, some are not. I designed this lightweight framework for test process assessment and improvement in order to focus the test team and test manager on a few test areas that they simply must do properly. This contrasts with other, more expansive, and more complex frameworks. In addition, the Critical Testing Processes (CTP) framework eschews the prescriptive elements of other frameworks because it does not impose an arbitrary, staged maturity model.
What’s the problem with prescriptive models? In our consulting work, we have found that businesses want to make improvements based on the business value of the improvement and the organizational pain that improvement will alleviate. A simplistic maturity rating might lead a business to make improvements in parts of the overall software process or test process that are actually less problematic or less important than other parts of the process simply because the model listed them in order.
CTP is a non-prescriptive process model. It describes the important software processes and what should happen in them, but it doesn’t put them in any order of improvement. This makes CTP a very flexible model. It allows you to identify and deal with specific challenges to your test processes. It identifies various attributes of good processes, both quantitative and qualitative. It allows you to use business value and organizational pain to select the order and importance of improvements. It is also adaptable to all software development lifecycle models.
Since CTP focuses on a small number of processes, how are those processes selected? What is a critical testing process? Here’s a quick way to think about it: If a test team executes the critical testing processes well, it will almost always succeed, but if it executes these activities poorly, even talented individual testers and test managers will usually fail. Let’s expand this definition a bit.
First, I defined a process as some sequence of actions, observations, and decisions. Next, I defined testing as the activities involved in planning, preparing, performing, and perfecting the assessing of the quality of a system. So, with a definition of a test process firmly in hand, what makes a test process a critical test process? I applied four criteria:
In other words, a critical test process directly and significantly affects the test team’s ability to find bugs, build confidence, reduce risks, and generate information.
Based on these criteria, I identified the following twelve critical testing processes:
You might notice that I’ve described each of these processes in terms of what an optimal process will achieve. If the process does not achieve those standards of capability—and more—then it is not optimal and has room for improvement.
How can you use CTP for assessment and improvement? Test process improvements using CTP begin with an assessment of the existing test process. This assessment will identify which of the twelve test processes are currently done properly and which need improvement. The assessment results in a set of prioritized recommendations for improvements. Whether you use the framework to do your own assessment or hire a consulting firm like RBCS to do it, the assessment should base the recommendations on organizational needs.
Since the assessments are tailorable, they can vary depending on the specific organization and its needs. We have done narrowly focused CTP assessments that looked only at one test team, we have done CTP assessments that looked only at a one or two test processes like the test system, and we have done broad CTP assessments that looked at everything that affects quality. So, while CTP assessments vary, we tend to examine the following metrics during a CTP assessment:
In addition, we also tend to evaluate the following qualitative factors, among others, during a CTP assessment:
Once an assessment has identified opportunities for improvement, the assessor will develop plans to implement this improvement. While the model includes generic guidelines for process improvement for each of the critical testing processes, the assessor is expected to tailor those heavily.
I designed the CTP model to be very flexible. It does assume the primary use of an analytical risk-based testing strategy, balanced with a dynamic testing strategy. However, you can adapt CTP to use other test strategies primarily, such as requirements based, checklist based, or model based.
There are some five to ten metrics for each critical testing process, along with some qualitative evaluations. Now, we don’t typically look at every metric for every process on every single assessment because the selected metrics, like the model itself, are tunable. This results in a customer-centric assessment.
So, what is the value of a CTP assessment? CTP, like any other process model, provides a starting point, a standard framework, and a way of measuring your processes. A process assessment using a process model identifies opportunities to improve your current process that could not be identified by applying continuous process improvement techniques, such as Deming’s Plan-Do-Check-Act (PDCA), to your current existing processes. This is true because techniques such as PDCA are incremental improvement techniques, while process models allow provide a method for quantum leaps in process improvement through the introduction of known best practices.
Properly done, CTP assessments deliver specific recommendations, along with the order in which to implement them. When my associates and I carry out CTP assessments for clients, we typically deliver a 50 to 100 page report with our assessment of the critical testing processes, our recommendations for improving them, the business justification for the improvements, and a roadmap for implementing those improvements. If you look at any two of our assessment reports, you might see very similar recommendations but in very different order. Why? Because each client has a different level of opportunity associated with the recommendation. In some cases, constraints or preconditions can influence the order. Those constraints and levels of opportunity tend to be unique from one organization to another, and a non-prescriptive model adapts to those unique organizational needs.
Let me close with an important note: Whether you use CTP or some other test process assessment model, don’t use process assessments as a one-time activity. Having done an assessment and found opportunities to improve, you should—of course—improve. Furthermore, at regular intervals, you should reassess to see the effect of the changes. Based on that reassessment, you should course correct your process improvement.
The process of negotiating a software testing budget can be painful. Some project managers view testing as a necessary evil that occurs at the end of the project. In these people’s minds, testing costs too much, takes too long, doesn’t help them build the product, and can create hostility between the test team and the rest of the development organization. No wonder people who view testing this way spend as little as possible on it.
Other project managers, though, are inclined to spend more on testing. Why? Smart software managers understand that testing is an investment in quality. Out of the overall project budget, the project managers set aside some money for assessing the product and resolving the bugs that the testers find. Smart test managers have learned how to manage that investment wisely. In such circumstances, the test investment produces a positive return, fits within the overall project schedule, has quantifiable findings, and is seen as a definite contributor to the project.
As Phil Crosby wrote, “quality is free.” This can be demonstrated by breaking down quality related costs as follows:
Conformance costs include prevention costs and appraisal costs. Prevention costs are moneys spent on quality assurance—tasks like code reviews, walkthroughs, or inspections, requirements definition, and other activities that promote good software. Appraisal costs include moneys spent planning test activities, developing test cases and data, and executing those cases—once.
Nonconformance costs come in two flavors, internal failures and external failures. The costs of internal failure include all expenses that arise when test cases fail the first time they’re run—as they often do. Think through the process: The tester researches and reports the failure, the developer finds and fixes the fault, the build engineer produces a new release, and the tester retests the new release to confirm the fix and to check for regression. The costs of external failure are those incurred when, rather than a tester finding a bug, the customer does. In these cases, not only does the same process described above occur, but you also incur the technical support overhead and the more expensive process of releasing a fix to the field rather than to the test lab. In addition, consider the intangible costs: Angry customers, damage to the company image, lost business, and maybe even lawsuits.
Two observations lay the foundation for the enlightened view of testing as an investment. First, like any cost equation in business, we will want to minimize the cost of quality. Second, while it is often cheaper to prevent problems than to repair them, if we must repair problems, internal failures cost less than external failures.
Let's look at a hypothetical case study, illustrated in the figure below. Assume we have a software product in the field, with one new release every quarter. On average, each release contains 1,000 “must-fix” bugs—unacceptable defects—which we identify and repair over the life of the release. Currently, developers find and fix 250 of those bugs during development, while the customers find the rest. Cost of quality accounting shows that, on average, each pre-release defect costs $10, while field failures cost $1,000. As shown in the “No Formal Testing” column in the figure below, our cost of quality is three-quarters of a million dollars—and customers are mad! So, we invest $70,000 per quarterly release in a manual testing process. The “Manual Testing” column shows how profitable this investment is. The testers find 350 bugs before the release, which cuts almost in half the number of bugs found by customers. Not only does this make the customers happier, but, because cost of quality accounting shows that we pay only $100 on average for each bug found by testers, our total cost of quality has dropped to about half a million dollars. Our ROI on this $70,000 investment is 350%.
Return on the Software Testing Investment:
In some cases, we can do even better. For example, suppose that we invest $150,000 in test automation tools, amortizing that investment over twelve quarterly releases, and manage to find about 40% more bugs? Finding 500 bugs in the test process would lower the overall customer bug find count for each release to 250—a huge improvement over the initial situation. In addition, cost of quality would fall to a little under $400,000, a 445% return on investment.
Cost of quality analyses on software process improvement bear out these figures. We have seen clients who enjoy return on the software testing investment that ranges anywhere from 50% to as high as 3200%. While software testing is only part of achieving software quality, it is an important part, and a substantial investment is justifiable to achieve such phenomenal gains.
To get started, you’ll need a management team wise enough to look at the cost of quality over the entire life of the software release. A management team that ignores the long term and focuses just on the budget required to get the software out the door initially does not see testing as an investment. Quality is given lip-service when the only priorities are shipping something—anything—on a given schedule and within a given budget.
However, having supportive, far-sighted management is only a necessary—not a sufficient—condition for achieving positive returns on your test investment. Just as in the stock market, there are right and wrong ways to invest. Picking the right tests, managing the appropriate quality risks, using the proper tools and techniques, and driving testing throughout the organization will result in optimal returns, while failure in any one of these areas can mean disappointing or even negative returns. These topics come up again and again in this blog, but if you'd like me to address one or more of them specifically, please let me know.
In order to design, develop, and run tests, you need what’s often referred to as a test oracle, something that tells you what the expected, correct result of a specific test should be. Specifications, requirements, business rules, marketing road maps, and other such documents frequently play this role. However, what if you receive no formal information that explains what the system under test should do?
In some organizations with mature development processes, the test department will not proceed without specifications. Because everyone expects to provide a specification to the test team as part of the development process, you are seen as reasonable and within the bounds of the company’s culture when you insist on written specs.
Trouble arises, however, if you stiffen your neck this way in a company that operates in a less mature fashion. Depending on your company’s readiness to embrace formal processes (and also on your personal popularity, tenure, and political clout), any one of a spectrum of outcomes could occur:
The moral of this story is that you should carefully consider whether your company is ready for formal processes before you insist on requirements or design specifications and other accoutrements of mature development projects.
This situation is exacerbated by the increasing popularity of Agile methodologies. Some of the people behind Agile approaches have endorsed a set of principles that include the following, “The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.” As you can imagine—if you are not already dealing with this situation—such a principle discounts the role of written specifications, encouraging instead an on-going dialog about the correct behavior of a system. In other words, the outcomes of discussions between stakeholders represent correct behavior. These discussions can happen in a meeting, but for some RBCS clients following Agile methods these discussions happen as one-on-ones between developers and another stakeholder.
I’m all for ongoing dialog between project stakeholders. However, unless there are written minutes from these discussions, agreed upon by all project stakeholders, including the test team, this approach has the risk that different people come to different conclusions about what the outcome of the discussion was. Of course, this is a big testing challenge if you weren’t part of the discussion.
Even more challenging is the need to deal with the possibility that at any point the definition of correct behavior can change. The Agile principles include this, saying, “Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.” While I can see the business value in being able to shape software like modeling clay up until the last minute, it’s hard to create and maintain test cases if the definition of correctness is continually re-defined.
When properly applied, Agile models can bring some advantages to the test team. However, they also bring challenges, not the least of which is the tendency to devalue solid, stable, documented definitions of correct behavior.
If you have to deal with a situation where the project team cannot or will not deliver written specifications in advance, during test development, you might consider the following options for testing without specifications:
If you are testing a commercial product, remember that you have the benefit of competitors. Because your customers will expect your product to behave substantially like the products of your competitors, these competitive products are, in a sense, your test oracle. In compatibility test labs, for example, most projects have a reference platform—a competitor’s system, against which the system under test is being positioned, in the hope of demolishing it in the marketplace.
If your technical colleagues won’t tell you what the product should do, perhaps your friends in sales and marketing will. In my experience, sales and marketing people live to create glitzy presentations showing where the product line is going. Although they can be general and imprecise, these documents might tell you which features and capabilities the product should support. If you’re testing a product for which questions about supported features are harder to answer than questions regarding correct behavior, these documents might suffice for a somewhat vague but useful oracle.
Ask your customer-support colleagues. Your colleagues in customer support might not have much information about what the product should do, but they probably know what they don’t want the product to do. Since your testing stands between them and the hellish scenario outlined in the previous section, they are usually happy to tell you.
Unless the product is truly unique, you can use inductive reasoning to figure out what constitutes reasonable expectations and correct behavior in many cases. The generic categories into which products fit tell you a lot about what the products are supposed to do: a word processor, a Web browser, a PC, a laptop, a server, an operating system. Some esoteric questions might arise, but a core dump, a system crash, a burning CPU, garbage on the screen, an error message in the wrong language, and abysmal performance are indisputably bugs.
If in doubt, you should consider any suspect behavior buggy. Because you don’t have a crisp way of determining pass and fail conditions, you will make mistakes in result interpretation. Remember that calling correct behavior a bug and working through the bug life cycle is less detrimental to product quality than failing to report questionable behavior that does turn out to be a bug. Be sure to file bug reports when questions arise.
One thing to keep in mind about this situation is that you are definitely not alone. Many people are struggling with the right amount of documentation to gather, and errors are made on both sides. I try to maintain an open mind, even though the 20-questions approach to defining expected results is somewhat frustrating. It is a good idea, if you’re working in a poorly specified situation, to make sure management understands that your test development will be less efficient due to the need to pull information from other groups. My usual rule of thumb is that the lack of clear requirements and design specifications imposes a 20- to 30-percent inefficiency on test development, and I estimate accordingly.
It's not just hype; it's government policy. That phrase could describe Malaysia's announced ambitions to become a software testing "hub." Hub seems to mean a preferred software test outsourcing destination, but it might also have other connotations that are less obvious.
In his opening remarks at the SOFTEC 2010 conference, government minister Mohamed Yakcop made the point again and again. Come to Malaysia (or stay in Malaysia) and help us build the country into a worldwide, premier venue for software test outsourcing.
It's a nice ambition for a country to have. As another fellow foreign testing dignitary who was sitting next to me as Yakcop spoke (and who shall remain nameless) asked after Yakcop's speech, "Can they pull it off?" The answer to that question depends on a number of factors. With 20+ years of experience with outsource testing, I can identify some key enablers to outsourcing success that help a country establish itself as a powerhouse:
I've been to Malaysia about a dozen times now over the last few years. From what I've seen, from the hundreds of people I've talked to and trained there, and from the leaders behind this hub concept, it certainly looks like most of the enablers are there. Malaysia is a place to watch in the software testing industry, and it would be unwise to bet against people like Mohamed Yakcop and Mastura abu Samah, President of the Malaysian Software Testing Board. To paraphrase Lenin, the Malaysians involved in this Malaysian Software Testing Hub initiative are software testers in a hurry.
As with any other activity, a major element of doing testing well is using the right people. The right test team can mean the difference between good software and bad software, with serious security, performance, or reliability problems. So, let’s look at how you can develop a test team that can help you achieve your quality goals in an effective and efficient manner.
I’ll start with how not to develop a test team. Many managers—especially those who have never seen professional testing—assume anyone can do it. They try to get the job done with junior programmers, users, or business analysts. While these people have a role to play, by themselves they do a poor job of testing. For example, our assessments show that a professional, trained test team typically finds over 85% of the bugs in the software that they test. Users, business analysts, and programmers are lucky to find 40% of the bugs. So, while we need programmers to unit test the code they write, and users and business analysts to acceptance test the applications as they go into production, most of the system testing should be done by a professional test team.
What skills should a professional test team have? This varies from project to project, but the skills fall into three groups. First are application domain skills, an understanding of the business problem solved by the application. Second are technical skills, an understanding of the technologies used to build the application. Third are testing skills, which are the skills that testers bring to the quality assurance role that neither programmers, business analysts, nor users can bring. Professional testers know how to analyze risks so that testing focuses on the right things. Professional testers can plan testing in a way that maximizes the value realized from the time and resources invested. Professional testers can apply test design techniques to find bugs that would otherwise slip past people.
True professional testers can be hard to find, so you might decide to develop your own team around a core of one or two senior testers. This is certainly a strategy that can succeed, and many of our clients have succeeded with it. Companies such as RBCS offer a wide range of training programs for testers throughout their career arc, from entry level to seasoned expert. As with other professions, there are mature, globally-recognized certifications available for testers. The best and most widely-accepted of these certifications, the International Software Testing Qualifications Board, offers Foundation and Advanced certifications as a way to measure progress towards professionalism throughout the testers career, through objective, vendor-independent exams.
You’ll need a particular set of skills if you want to use automation for your testing. Test automation tools allow you to automate important activities like regression, performance, and reliability testing. Regression testing checks whether changes to your application broke existing features. Performance testing checks for slow response times under various likely levels of usage. Reliability testing checks for frequent or lengthy application availability problems. It takes years to learn how to do test automation properly. Our clients have told us horror stories about losing hundreds of thousands of dollars on failed test automation projects, so make sure that anyone you hire for such a position has been through a lot of test automation projects.
You might also need people with testing experience in your application domain or industry. Some of our clients, such as those in medical systems and banking, find that regulatory compliance issues associated with their application have important implications for testing, implications known only to experienced testers who have previously tested such applications. Other clients, such as those in video games and oil exploration applications, find that special knowledge of how the applications work is essential. That said, don’t over-estimate the value of application domain knowledge, because that route often leads people to employ business analysts and users exclusively, with the poor outcomes I mentioned before.
A proper, professional test team has the right mix of skills and experience for the complex and often under-appreciated job of checking the quality of the applications. Not only will such a team often find twice as many bugs as one composed of non-testers, but they will also typically save their employers the cost of employing them, many times over. Our assessments have shown that professional test teams save anywhere from 40% to 3200% more than the team costs, primarily through avoided costs of production failure. Armed with the proper test team, you are ready to achieve your quality goals, and to save money while doing so.
While I focus on immediately applicable ideas in this blog, every so often, maybe a little light-hearted fun is in order? On that note, here's short video of me ring a big bell in Nanking, China. Let me know if you like it! Ringing Confucius Bell in Nanking
It's a well-known--though not always widely practiced--best practice of software engineering to build quality into an application by integrating multiple quality assurance activities into the entire software development and maintenance process. This is important, but most of us don’t have the luxury of thinking about only one application. Today, it’s not enough to know that each application works properly on its own. More and more, everything talks to everything else, and the trend is for greater interoperability in the future. Before release, we need to know that these disparate applications will work together.
Of course, you can and should expand the scope of testing (and other quality assurance activities) to address issues of interoperability. Requirements engineers should consider the various applications that should talk to each other. Designers and architects should construct simple, robust channels for data and control to flow between those interoperating applications.
The whole team should particularly consider security, performance, and reliability risks associated with interoperability. Connecting two or more applications together magnifies the potential likelihood and impact of security vulnerabilities. For example, a previously-standalone application with access to customer credit card information, when connected to an application that provides information to your company web site, might create an avenue for leakage of this sensitive data.
When applications communicate, that can affect the performance of each application. For example, suppose an application can now request data from another application that stores the requested information on two or more tables. That could result in a database join query that deals with large volumes of data. This problem can be made worse yet if the table indices are not set up properly. I once worked with a client where a multi-year project that involved dozens of people failed after three years of effort because of database-related performance problems.
Application interoperability can also affect reliability. For example, if delays or losses of information occur, this can result in timeouts, unexpected voids in data records, and processing with illegal default values. In addition, the lack of proper data conversion by the sending or receiving application can occur. This could cause the application to stop responding or even crash, or simply result in “garbage in garbage out” scenarios that cause failures later. For example, the Mars Climate Orbiter mission was lost due to problems with data conversion between metric and English units in two interoperating NASA systems.
As I mentioned, the project team should start to address these risks during requirements definition and design. In addition, code reviews and static analysis can also locate potential problems. However, you’ll also need to carry out system integration testing prior to releasing interoperating applications.
When you do system integration testing, interoperability is one of the main types of testing you are typically doing, along with security, performance, reliability, and end-to-end functionality. All of these types of testing require the use of realistic test data so that important test conditions are covered. Production data is the obvious place to obtain realistic test data, but there are pitfalls and surprises that can occur.
Remember that production data can include personal and confidential information. For good reason, organizations tend to place restrictions on who can access such data. This becomes particularly critical if you intend to outsource some of the system integration testing. There are tools available to anonymize production data in a way that preserves valuable test conditions while irreversibly hiding personal data. Production data sets are often very large, so be sure to allocate plenty of time to complete any test data anonymization project.
Keep in mind that using software as a service (SAAS) and open source applications changes the situation, but it certainly does not make these risks go away. With either approach, if you are sharing data with such application, you lose some of the control you would otherwise have over the interfaces between those applications and your own. In the case of open source software, an organization can at least determine its own schedule for taking updates. For SAAS, the software can change without any warning. If your application shares data with open source or SAAS applications in typical ways, you can rely (to some extent) on the risks being addressed as part of the release process. However, if your application interoperates with these applications in an atypical fashion, you could end up discovering issues that no one thought to address. You should try to understand whether your usage of these applications matches the typical usage in order to understand interoperability risks and thus the degree of system integration testing required.
Today, with communicating systems both collocated and distributed across the cloud, interoperability matters more than ever. Project teams should start considering interoperability during requirements and design. They should continue to address key interoperability risks such as performance, security, and reliability throughout the lifecycle, including code reviews, static analysis, and testing, especially during system integration testing. Since system integration testing involves test types that have very particular test data requirements, be sure to plan carefully for this phase of testing. And, in your system integration test plans, don’t forget to include open source and SAAS applications that you use in your data center. While managing interoperability risks is not trivial, it is essential to avoid nasty interoperability surprises.