A well-written script can literally be executed by anyone, but requires a lot of test design time. A poorly written script can just be a disaster. A compromise is to use test charters, either alone, or in conjunction with test scripts. A test charter is a focused test of a specific application functional area, written in a narrative style. While test charters require that the executor have more knowledge about the system, they make it much easier to create and maintain a robust test suite.

Test charters have several advantages:

• If an application function has many similar paths that need to be tested, it is easier to write one script and write a number of charters to reuse the script to test the various paths.
• Charters support testing edge cases or special data cases (negative numbers, boundary tests, invalid or odd inputs) by only requiring that you tell the tester what type of data to enter, and where they need to enter it.
• Because charters are not reliant on navigation, they are less likely to require rewriting.
• When testers have new ideas for testing, they just have to write a short paragraph for a charter, rather than go through the work of an entire new script. This makes it more likely they will write them down.
• A series of charters acts as mental stimulus for the next tester to add their own ideas.
• Charters support later automation by providing more cases that need to be tested.

A simple exercise: imagine if you were going to test Google searches. Would you want to write one script for every single possible test (PageRank tests, inserted Google Map results, filter tests, suggested searches for typos, etc.)? Of course not. You’d end up with a library of hundreds or even thousands of scripts that you’d be forced to maintain whenever the design changed.

Instead, you could write charters that would serve for initial manual testing, and then an automated tester could follow up behind you to use the best charters to build a robust automated regression library. Examples:

• Execute a search for selected companies in the Fortune 500. Verify that relevant results are returned for the company. (Automation could easily expand this to the Fortune 1000 and return deeper results, with analysis to catch Google bombs, irrelevant hits, or unwanted hits).
• Execute a search for a selected group of top searches. Verify that Google News results are returned along with standard results. (Automation could add analytics to see the average age of news results, or reliability of news links.)
• Execute searches using a selection of non-English characters from various languages. (Automation would test words using all possible non-English characters.)

If you’re finding that your testing has holes, or that the work you are putting into test design isn’t yielding the results you want, create a suite of test charters for just one piece of an application, and watch the ideas pour forth.

Cem Kaner, a professor at Florida Tech, has some great information on exploratory and charter testing.

Ed Cook is a 12-year veteran of IT, transitioning help desk experience into software test analyst experience for application projects in several industries. Ed has provided custom help desk services for Unisys and expert functional test solutions for the State of Indiana’s Department of Child Services and Sallie Mae on a wide range of in-house applications. Currently, he is a software testing consultant here at Fusion Alliance.

Many of our existing customers and some potential new customers come to Fusion Alliance wanting to know more about test automation and if it’s right for their business.  To answer that question, the following assessments must be done to determine the rationale for or against using test automation.

1. Assess the maturity level of the customer’s quality control/quality assurance departments and their processes.
2. Assess the nature of the engagement Fusion Alliance is being asked to assist the customer with.

These assessments are essential, as many organizations make the mistake of thinking that test automation will immediately reduce the costs of testing, increase test coverage, and shorten test cycles. These are sometimes the longer-term benefits of test automation, but they are not immediately evident. To determine if test automation will indeed reduce costs for the organization in the long term, the ROI must be calculated.  The ROI for test automation is simply the benefit of test automation divided by the cost of test automation.

The cost of test automation is higher than manual testing and should include any costs related to hardware, software, and licenses; time for resources to produce scripts; and cost of the resources themselves.

The benefits of test automation must be calculated over a period of time for the software under test and take into account the reduced time to execute tests and the ability to test as frequently as the organization wishes.  These figures should be compared to the costs of manually testing the same software.

Think of the cost of test automation vs. the cost of manual testing as follows:

Cost of test automation = price of hardware needed + price of  tool required + time to develop test scripts + (time to maintain test scripts x number of times test scripts are executed) + (time to execute test scripts x number of times)

Cost of manual testing = time to develop test scripts/charters + (time to maintain test scripts/charters x number of times tests are executed) + (time to execute test scripts/charters x number of times)

After figuring the costs of test automation and manual testing, calculate ROI as follows:

ROI = (cost of manual testing – cost of test automation) / cost of test automation

By now, you might be saying to yourself “that seems too easy.”  If you are, you would be correct.  Other factors must be taken into account, such as the benefits of manual testing over test automation.  The most complex parts of the software should be left to a manual tester to analyze to ensure maximum quality.  Context must also be taken into consideration. What is the complexity of the software? What is the size of the software and what is its role in the business? Smaller, less complex applications that won’t take a considerable amount of time to test won’t require test automation because the upfront costs will outweigh the benefits gained. Also keep in mind the types of testing you are looking to automate. Each of these different types will have a different ROI that needs to be taken into consideration.

Thinking about all of these things upfront will not only help you determine whether to automate your testing or not, but also help you plan your overall testing effort and give you a jump start on your test plan(s).  For help performing these assessments and getting answers to your questions, please contact our knowledgeable consultants.

Josh is a Software Test Analyst at Fusion Alliance. He is eager to learn new technologies and share what he learns with others. Josh keeps up on the latest in software testing by reading articles and blogs. Josh also maintains his own blog on software testing. Josh earned his undergraduate degree in English Studies from Ball State University and is a member of AST (Association for Software Testing) and IWST (Indianapolis Workshops for Software Testers).

Well…the answer is: “It depends.” Testing is not some trivial activity that should be performed in excess in order to ensure quality. More testing does not necessarily mean higher quality. Remember: We cannot test quality into a system.

According to the Certified Software Test Engineer (CSTE) Certification Book Of Knowledge, the act of testing is one of three categories of cost associated with producing a quality product. This category is called the “Appraisal Cost” and is defined as:

“Money spent to review completed products against requirements. Appraisal includes the cost of inspections, testing, and reviews. This money is spent after the product is built but before it is shipped to the user or moved into production.”

As mature software testing professionals, we should make sure we have an even better understanding of how testing dollars are spent than our management does. I say “better” because we are the ones in the trenches and this understanding will enable us to make and communicate better testing decisions as the project evolves. Testing is a service, and it is our responsibility to always ensure that our testing is providing value to our customer.

Testing is inherently expensive. Couple that with the current economic climate, and you have a formula for companies that now, more than ever, are watching how capital dollars are being spent.  At the time of this writing, I’m on assignment where the client had already identified the testing scenarios they wanted us to write scripts for and our testing strategy. During execution, I was able to step back and take a broader view of our current testing approach. I started a discussion, which led to a chain of events that ultimately caused us to revise our testing approach and, as a result, spend our client’s testing dollars more efficiently. This begs the question, “So how did I know I needed to stop testing?” Below is a list of heuristics that I have learned over the course of my career to help me identify when I might need to stop testing.

1. Is your testing engaging?

This may seem trivial but I assure you it is not.

Testing is a highly cognitive activity that requires mental stamina. Without adequate attention you immediately compromise the efficacy of whatever tests you are executing. Testing also requires a degree of curiosity that should drive questions you ask of yourself while testing. Participating in an engaging test activity will hold your attention and spark curiosity.

2. Is your approach still viable?

This is a question you should ask yourself continually during test design and execution. As professional testers, we must always account for variable change. What once was a valid test may no longer be, due to changes in the system under test, changes in project scope, and budget. Your ability to assess the viability of a test is critical in determining whether or not to stop testing.

3. What does your gut tell you?

Wikipedia.com has a definition for intuition that I just love: “understanding without apparent effort.”

Intuition is a subject matter that continues to be discussed within the science community. Your inability to articulate the why of an idea doesn’t negate its validity. Lesson 26 of Lessons Learned in Software Testing states: “Intuition is a great beginning but a lousy conclusion.” Be comfortable in what your gut is telling you. Just remember it’s a beginning. Realize that you must anchor your idea with justifiable evidence or risk being viewed as a loon. If you find yourself answering yes to only one of these questions, I would simply make a mental note and reassess the following business day. Everyone has a bad day now and again. However if you find yourself answering yes to two or more of these, I would recommend you stop testing and start a discussion.

This brings me to my second point: the value of communication. I’m a strong advocate of communicating early and often. Start the conversation with your fellow testing colleagues. Explain to them why you believe your current testing approach fails to provide the value previously agreed upon. Be sure to include possible solutions to address this shortcoming in the discussion. Include as many individuals necessary to ensure you take advantage of the wide and varying experiences represented within your team. Once you confirm that your concerns and proposed solution is valid, you should formally communicate this information to the necessary individuals in management.

By communicating your concerns sooner, you increase the odds of obtaining management support as well as any additional resources and/or related tools. The time to let your customer know that you have concerns about the value of a testing approach is before you spend their testing dollars, not after.

In an attempt to drive this point home let me leave you with this thought: Have you ever spent money on an auto repair only to have the mechanic tell that the original issue still remains? Rest assured that when you engage Fusion Alliance for your testing needs, you immediately gain access to a group of expert and passionate testing professionals who collectively represent 200+ years worth of software testing experience. You will work with individuals who are conscious of how our clients are spending their testing dollars. Fusion Alliance testing professionals are committed to continually assessing the effectiveness and efficiency of your testing efforts throughout the project life cycle.

Ready to spend your testing dollars wisely? Contact us.

Bobby Washington is an IT professional with 7 years of experience in Information Systems with concentration in both manual and automated software testing. Bobby has worked to embed QC within an organization’s software development life cycle by facilitating and implementing testing best practices.  Life cycle activity involvement includes requirements analysis/elicitation, development and maintenance of test plans/scripts, test automation, test execution, and post-production tasks. He has worked in the insurance, retail, and access control security industries on platforms ranging from IBM AS400 mid-range systems to virtualized servers via VMware.  When he isn’t reading and studying up on new schools of thought related to the practice of software testing, Bobby can be found enjoying his family or casually playing a video game.