Our notable partnerships give us access to the best resources and tools for the job.
Get software testing training resources:
Automation typically refers to test automation, which is the use of software to perform or support test activities, e.g., test management, test design, test execution and results checking.
One of our clients, CA, has continued to impress us with innovative ways to go about their testing. An upcoming software testing article will discuss how we are helping them institute risk-based testing. Read this article to learn how one of their teams is using a leading-edge open source testing tool, WATIJ, to help contain regression risk.
You're familiar with test automation, but what is a dumb monkey? How can it help you automate your testing and explore very large screen flows and input sets? Is it true that you can build dumb monkeys from freeware with no tool budget? What kind of quality risks can dumb monkeys address? Read this article to learn the answers to these and other test automation questions.
Many of us got into technology because we were fascinated by the prospect of using computers to build better ways to get work done. (That and the almost magical way we could command a complex machine to do something simply through the force of words coming off our fingers, into a keyboard, and onto a screen.) Ultimately, those of us who consider ourselves software engineers, like all engineers, are in the business of building useful things.
Of course, engineers need tools. Civil engineers have dump trucks, trenching machines, and graders. Mechanical engineers have CAD/CAM software. And we have integrated development environments (IDEs), configuration management tools, automated unit testing and functional regression testing tools, and more. Many great testing tools are available, and some of them are even free. But just because you can get a tool, doesn’t mean that you need the tool.
When you get beyond the geek-factor on some tool, you come to the practical questions: What is the business case for using a tool? There are so many options, but how to I pick one? How should I introduce and deploy the tool? How can I measure the return on investment for the tool? This article will help you uncover answers to these questions as you contemplate tools.
The Testing performed by our Test Group is Black Box Testing. This paper will discuss the following areas of our test effort in more detail:
Automated test tools are powerful aids to improving the return on the testing investment when used wisely. Some tests inherently require an automated approach to be effective, but others must be manual. In addition, automated testing projects that fail are expensive and politically dangerous. How can we recognize whether to automate a test or run it manually, and how much money should we spend on a test?
This is an excerpt from my book, Expert Test Manager, written with James Rommens and Leo van der Aalst. I hope it helps you think more clearly about the test strategies you use.
A test policy contains the mission and objectives of testing along with metrics and goals associated with the effectiveness, efficiency, and satisfaction with which we achieve those objectives. In short, the policy defines why we test. While it might also include some high-level description of the fundamental test process, in general the test policy does not talk about how we test.
The document that describes how we test is the test strategy. In the test strategy, the test group explains how the test policy will be implemented. This document should be a general description that spans multiple projects. While the test strategy can describe how testing is done for all projects, organizations might choose to have separate documents for various types of projects. For example, an organization might have a sequential lifecycle test strategy, an Agile test strategy, and a maintenance test strategy.
[Note: This is an excerpt from Agile Testing Foundations: An ISTQB Foundation Level Agile Tester Guide, by Rex Black, Marie Walsh, Gerry Coleman, Bertrand Cornanguer, Istvan Forgacs, Kari Kakkonen, and Jan Sabak, published July 2017. Kari Kakkonen wrote this selection. The authors are all members of the ISTQB Working Group that wrote the ISTQB Agile Tester Foundation syllabus.]
The traditional way of developing code is to write the code first, and then test it. Some of the major challenges of this approach are that testing is generally conducted late in the process and it is difficult to achieve adequate test coverage. Test-first practices can help solve these challenges. In this environment, tests are designed first, in a collaboration between business stakeholders, testers, and developers. Their knowledge of what will be tested helps developers write code that fulfils the tests. A test-first approach allows the team to focus on and clarify the expressed needs through a discussion of how to test the resulting code. Developers can use these tests to guide their development. Developers, testers, and business stakeholders can use these tests to verify the code once it is developed.
A number of test-first practices have been created for Agile projects, as mentioned in section 2.1 of this book. They tend to be called X Driven Development, where X stands for the driving force for the development. In Test Driven Development (TDD), the driving force is testing. In Acceptance Test-Driven Development (ATDD), it is the acceptance tests that will verify the implemented user story. In Behaviour-Driven Development (BDD), it is the behaviour of the software that the user will experience. Common to all these approaches is that the tests are written before the code is developed, i.e., they are test-first approaches. The approaches are usually better known by their acronyms. This subsection describes these test-first approaches and information on how to apply them is contained in section 3.3.
Test-Driven Development was the first of these approaches to appear. It was introduced as one of the practices within Extreme Programming (XP) back in 1990. It has been practiced for two decades and has been adopted by many software developers, in Agile and traditional projects. However, it is also a good example of an Agile practice that is not used in all projects. One limitation with TDD is that if the developer misunderstands what the software is to do, the unit tests will also include the same misunderstandings, giving passing results even though the software is not working properly. There is some controversy over whether TDD delivers the benefits it promises. Some, such as Jim Coplien, even suggest that unit testing is mostly waste.
TDD is mostly for unit testing by developers. Agile teams soon came up with the question: What if we could have a way to get the benefits of test-first development for acceptance tests and higher level testing in general? And thus Acceptance Test-Driven Development was born. (There are also other names for similar higher-level test-first methods; for example, Specification by Example (SBE) from Gojko Adzic.) Later, Dan North wanted to emphasize the behaviours from a business perspective, leading him to give his technique the name Behaviour-Driven Development. ATDD and BDD are in practice very similar concepts.
Let’s look at these three test-first techniques, TDD, ATDD, and BDD, more closely in the following subsections.
[How can dumb monkeys built from free tools help you? Give this article a read to see a case study. Originally published in Software Testing Professional magazine in 2008, these ideas and techniques are still relevant to SDETs, Technical Test Engineers, and Technical Test Analysts looking to build their own automation solutions using open-source components.]
Arrowhead Electronic Healthcare has been creating eDiarys on handheld devices since 1999. Arrowhead helps pharmaceutical research and marketing organizations document important information about how their products are being used in patients’ homes.
ePRO-LOG is Arrowhead’s third generation eDiary product. The primary design goal of ePRO-LOG is to be able to rapidly deploy diaries used for data collection in clinical trails and disease management programs.
A typical diary may include 100 forms translated in 15 or more languages, and used in several locales. This results in a large number of software builds and configurations. As a result, we needed an automated test tool to address potential risks and to automate common tasks.
The most important quality risks we wanted to address were:
We needed an automated test tool with the following capabilities and features:
This is a case study in how we reduced our risks and achieved our test automation objectives in just a few months on a total tools outlay of $0.
Recorded February 16, 2017
Recorded September 7, 2016
Recorded May 19, 2016
Recorded March 30, 2016
Recorded March 3, 2016
Recorded August 7, 2015
Recorded July 7, 2015
Length: 1h 11m 9s
Free tools are great, because they are always in budget, right? Well, not always, because your time is valuable. You can waste a lot of time if you misuse free tools. However, with commercial tool vendors pounding down your doors, trying to convince you to buy their gee-whiz test tool (often with a gee-whiz price tag to match), you might forget to look at the open-source test tool options available to you. In this talk, drawn from three decades of industry experience including recent consulting work with many clients, Rex will provide some case studies—and the lessons we can learn from them—in successful use of free tools. He’ll also give some cautionary tales of free tool usage gone astray.
Length: 1h 30m 0s
Let’s suppose you bought a car. Six days later, someone from the dealership let himself into your garage, removed the tires on the car, installed some “updated” tires that actually had holes in them, and then left. In the morning, your car was there in the garage, all sad and undriveable on its flat, flabby tires. That’s clearly unacceptable, in fact even criminal, but we allow the same thing to happen all the time with software. Why? In this webinar, Rex will catalog infamous automated software updates, released without sufficient testing to wreak havoc, or at least inconvenience. He’ll then give a detailed roadmap for reducing your chances of being part of the problem.
Some people use the terms “verification” and “validation” interchangeably, but there are significant differences between them. Some people disparage verification, or deny that it’s even involved in testing. However, you can’t adequately build confidence and reduce risk in the software you test without using the proper mix of both. In this webinar, Rex will clarify the meaning of these two terms, give examples, and explain why both are essential to proper software testing.
Length: 0h 38m 45s
If you’ve been testing for any length of time, you know that the number of possible test cases is enormous if you try to test all possible combinations of inputs, configuration values, types of data, and so forth. It’s like the mythical monster, the many-headed Hydra, which would sprout two or more new heads for each head that was cut off. Two simple approaches to dealing with combinatorial explosions such as this are equivalence partitioning and boundary value analysis, but those techniques don’t check for interactions between factors. A reasonable, manageable way to test combinations is called pairwise testing, but to do it you’ll need a tool. In this inaugural One Key Idea session, Rex will demonstrate the use of a free tool, ACTS, built by the US NIST and available for download worldwide. We can’t promise to turn you into Hercules, but you will definitely walk away able to slay the combinatorial Hydra.
The Advanced Test Automation Engineer Boot Camp, created by Rex Black, past President of the International Software Testing Qualifications Board (ISTQB), past President of the American Software Testing Qualifications Board (ASTQB) and co-author of a number of International Software Testing Qualifications Board syllabi, is ideal for testers and test teams preparing for certification in a short timeframe with time and money constraints.