Blog

In my previous post, we discussed areas of test automation that may provide long-term reusable positive results within a manageable amount of time.

Examples:

• Check each entry field that has maximum-length string data.
• Check each required field that has null data.
• Challenge type-checking validation of each field that has data of a different type from what is allowed.
• Test for all data values that are simultaneously maximized or most stressful.
• Verify that the nominal value of data is accepted, stored, retrieved, and displayed.
• Logon/Logoff
• Password Validation
• Open/Close/Maximize/Minimize

Let’s discuss the next steps.

Expansion and Improvements

Test-design specification templates should be expanded beyond the typical empty-document sections to fill in to include generic validation rules, test-case checklists, common failure modes, and libraries of generic tests.  The templates should include many of the usual tests for application features, so that they do not have to be added or reinvented. Part of this generic design should implement “boilerplate” tests that will be the first ones that every testing effort should include, even before considering more sophisticated and instance-specific tests.  In this way, the generic test-design specifications allow any tester—for example, a new contingent staff member—to perform a range of generic tests, even before any customized, application-specific test planning.

Examples:

1. Identify a record or field upon which to operate (based on input name and parameter info).
2. Verify nonexistence.
3. Add item.
4. Read and verify existence of identical unchanged data.
5. Modify and verify matching modified data.
6. Delete and verify removal of item.
7. Identify item with type characteristics (for example, a data field) at an abstract level. This should not be limited to simple data types, but should include common business data types (for example, telephone number, address, ZIP code, customer, Social Security number, calendar date, and so on).

Mature examples:

1. Enumerate the generic business rules that are associated with the type.
2. Define equivalence partitions and boundaries for the values for each business rule.
3. Select test-case values from each equivalence class.
4. Generate randomized item from equivalence classes.

Further Advances

• Reduce ad-hoc reinvention of test designs by capturing and reusing common test patterns.
• Recognize and abstract test issues. Capture them in a form that can be reapplied as a higher-level chunk of activity instead of always dropping down to the detailed instance level.
• Package the results and experience of past test-design work into a reusable deliverable that can be applied to similar test efforts.
• Create checklists of generalized “things to test” that will be a resource for reuse on future versions or applications.  These generalized tests should augment the separate list of feature details (business rules) and associated tests that truly are unique to a specific project.
• Catalogue common and generic failure modes as items to verify, and make them part of test-design templates.
• Collect and document explicit test coverage categories, so that they can be incorporated into analysis and test design.
• Ensure that each category contains as complete a set of tests as possible to ensure coverage of the category.  Divide the results into generic and unique subsets of tests.  Reuse the generic portions of these lists for future efforts.
• Test-design specification templates should be expanded beyond empty-document sections to fill in to include generic validation rules, test-case checklists, common failure modes, and libraries of generic tests.

Conclusion

Testers should build test designs around reusable test patterns that are common to a large number of application-testing problems. The inclusion of these patterns as a standard part of the test-design template reduces the time that is spent on test design by eliminating the need to start with a blank test-design specification. Reuse of these patterns has the added benefit of codifying expert-tester knowledge, so as to increase the likelihood of catching common failures.

Final Words

The introduction of automated testing is in itself a project. Treat it so. Plan it as you would any major project. Make it a first-level level item.  Involve all members of your team from developers to management. Expect discussion, setbacks, and a few hefty headaches as you grow.

SOUND OFF: Have you implemented automated testing in your organization? What hurdles did you face?

Over the past few years, we’ve seen a considerable amount of verbs and nouns directed at the topic of automation testing. As time passes, more and more companies have been fascinated by the coolness of automated testing, perhaps sometimes neglecting the main reasons for it and the base requirement behind it. This has raised the question for us: is automation testing just a catch phrase or reality?

Automation Testing is a Component of the SDLC

The answer to this evidently relies on how any company is using automation testing within their software development life cycle (SDLC). It is important to evaluate this factor because automation testing itself is an entity of the SDLC. There must be proper understanding of what piece within the software will be automated and how much benefit it will provide after spending X amount of time behind automating that functionality of the software application.

I have seen many companies captivated by the awesomeness of automation testing and pursuing the delusion of being able to automate the testing process, but forgetting to evaluate what piece needs to be automated, what benefit it will provide, and mainly the process to determine the long-term goals of the automation testing process. Besides this, the main factor that can turn any automation testing into a catch phrase is choosing inappropriate testing tools, having too complex a framework, and not knowing the restrictions that could affect your automation testing process. One major constraint is knowing the time spent to keep those automated scripts updated and what the impact on those scripts will be if the software application’s functionality or user interface is changed.

Carefully Consider All of the Factors

To avoid automation testing being just a catch phrase, any automation engineer needs to evaluate the above mentioned factors. Keep in mind how much manual process any particular automated script will save and what will be the challenges in case the system functionality or the application’s user interface is changed. Also, it is very important that the automated testing tool support different OS versions and if you’re willing to test the application on multiple platforms and different OS versions.

There is no doubt that automation testing provides a great deal of benefits and saves a good amount of time—but only if it’s implemented in an appropriate manner, evaluating all factors mentioned above. However, the process of automation testing doesn’t stop here. What we have talked about so far is the input and having automation testing implemented for any organization. But what else after this is achieved?

There should be a unified process for reporting errors and tracking bug defects so that the results of your automation test can be easily incorporated within the bug-reporting tool. This is essential to track the progress of your software development, as well as the process of solving the identified bugs while performing testing. Using both of these processes, managers can easily view and track the status of their product development.

Lastly, to make automation testing a reality depends on the team responsible for its implementation. And we all should remember that we are as good as our team is… So make that catch phrase a reality and enjoy the goodies of automation testing.

Sound Off: Has your organization integrated automated testing holistically? Or is automated testing a passing fad?

Written in collaboration with Travis Moser, Experience Architect

Setting up Google Analytics to capture site metrics is usually a pretty straightforward task. However, sometimes it requires a bit of creative thinking when you run into unexpected challenges.

Graphic by David Vincente. Some rights reserved.

The Challenge

On a recent project, we were installing Google Analytics to capture data for a website that was contained within a sub-directory of a larger corporate site. The business owner wanted to capture data specific to her sub-site—in particular—how many visits to the sub-site it took for a user to submit a contact form. It was important for the business owner to know this to make a judgment on how well the content was doing to explain the offering and persuade the user to take the next step. A reasonable request, right?

Here’s where the challenge came in.  Because marketing campaigns were going to be sending visitors directly to the sub-site, the business owner needed to record visitors to the sub-site separately from visitors to the larger corporate site.  Currently, the cookie tracking unique visits was being written to the corporate domain. So, for example, if a user visited the corporate site two times, then visited the sub-site twice and submitted a contact form on the second visit, Google Analytics would show that it took this user four visits to submit a contact form. We would have no way of knowing how many of those visits prior to the last visit were actually visits to the sub-site.

We called on Experience Architect Travis Moser to do a little investigating.  Here’s what he learned and how he recommended resolving the problem.

The Solution

We needed to figure out how to show that even if visitors had previously visited the larger corporate site, they should be counted as new visitors when they arrived on the sub-site. At the same time, we needed to track these visits to the sub-site to the larger corporate site Google Analytics account. Essentially, we needed to track the visitors to the sub-site twice—once to the larger corporate Google Analytics account—and once to the sub-site Google Analytics account, while making sure the data for one account didn’t affect the data for the other.

On the surface, that doesn’t seem too difficult. Let’s say the domain name for the site is http://www.acme.com and the sub-site is at http://www.acme.com/sub-site/. After some research, this is the code that we found others using:

var _gaq = _gaq || [];
gaq.push(
  ['setAccount', 'UA-XXXXA-X'], // global site account number
  ['trackPageview'],
  ['ss.setAccount', 'UA-XXXXB-X'], // sub-site account number
  ['ss.trackPageview']
);

The problem with this code is that because both site accounts use the same domain name, their cookies will conflict, resulting in inaccurate visitor counts. The accuracy of visitor counts was critical to determining the effectiveness of the site content. We realized we needed to force Google Analytics to set two separate cookies—one for each account. This can be accomplished by either setting a separate domain name for each account or by setting the cookie path for one of the accounts to a different directory. Because the sub-site is in a separate directory, it made more sense to set the cookie path for the sub-site account like this:

 var _gaq = _gaq || [];
gaq.push(
  ['setAccount', 'UA-XXXXA-X'], // global site account number
  ['trackPageview'],
  ['ss.setAccount', 'UA-XXXXB-X'], // sub-site account number
  ['ss.setCookiePath', '/sub-site'], // different cookie for sub-site
  ['ss.trackPageview']
);

Now the data is tracked into two separate Google Analytics accounts with two separate cookies: the larger corporate site account’s cookie domain is set to www.acme.com, and the sub-site’s cookie domain is set to www.acme.com/sub-site/.  Problem solved! The business owner will have an understanding of how the new site content is doing to get visitors to take action, and the corporate site owners will still be able to accurately track unique visitors to the site overall.

Travis, you rock!

Follow the Rules of Software Development

Previously I spoke of the problems in attaining a level of success when implementing automated testing within a project.

For example:

• Spare time test automation (team members consumed by larger tasks)
• Lack of clear goals (is implementation of successful  automation processes the long-term goal or simply a wishful thought?)
• Lack of experience (inexperienced staff who may create short-term fixes without long term success)
• High team member turnover (when test automation is not its own project and team members better utilized elsewhere, or they simply run out of time to accomplish the automation goals)
• Reaction to desperation (struggling to learn and meet timelines simultaneously)
• Reluctance to think about testing (focus on automation prized above the testing itself)
• Technology focus (do the results meet the testing needs?)
• Labor intensive (management of the automation processes may prove to be far more extensive than manual testing if not created and managed as reusable resources)

This leads me to the conclusion that Test Automation should be looked at as a Software Development Project.

An organization is often oblivious to the fact that it is actually developing software; there is no distinction between the users and the developers. This is the place that test automation often finds itself.  Thus, dedicating resources to test automation and treating it like a development activity elevates it to the first level. This is the core of the solution to the problems of test automation. Test automation projects should be run just as any other software development project.

Like other software development projects, developers should be dedicated to developing the test automation.

• Test automation automates a task in which the developer is probably not an expert. Therefore, skilled testers should be consulted and should provide the requirements.
• Test automation benefits if the approach is designed before coding is started.
• Test automation code needs to be tracked and secured. Therefore, source code management is a requirement.
• Test automation will have bugs. Therefore, it is a necessity to have to plan to track them, test for them and provide change management.
• Users will need to know how to use the tests and the tools. Consequently, create and update user documentation.

Assuming that you are part of a software organization that already has some idea as to what reasonable and effective methods should be used for developing software; you are urged to abide by those rules established for software development in your own test automation. The goal is to maintain the concepts used for software development projects, alongside the considerations and challenges that are particular to test automation.

Examples:

1. Improve the testing process
2. Define requirements
3. Prove the concept
4. Encourage product testability
5. Design for sustainability
6. Plan for deployment

Intro to Automated Testing 101 (avoid initial high-end costs, training, and time investments)

As stated above, some of the most immediate goals of automated testing are as follows:

1. Improve speed and efficiency
2. Establish reliability of process
3. Create repeatable processes
4. Manage reusable processes

Additional objectives are:

1. Reduce costs
2. Prove the concepts
3. Improve the overall testing process

What efforts can be most quickly served through automation and positively affect the items listed above?

There at least two (2) primary conditions where the introduction of automation can create immediate cost and time savings with reduced training and lower initial financial investment.

1. Changes of environment, testing across multiple browsers (IE, Chrome, Firefox, Safari, etc.) or multiple browser versions (IE7, IE8, etc.).
2. Testing of systems that process large volumes of data i.e., retail, medical, research, etc.

In both of these cases, the tests remain essentially the same through each repetition. They can be created by recording keystrokes and mapping them to a script or by loading diverse data sets to test the flexibility of the system.

Test engineers start with a blank page in a test-design specification and fill it in detail by detail, when many of these issues are generic and common to nearly every application-testing scenario. Again, the introduction of recordable testing tools to create an initial package of test designs without programming and with minimal training is possible. These designs can be readily modified with the edit tools provided by relatively inexpensive testing suites (5 figures versus 6 to 7 figures) to adjust designs quickly for reuse.

Examples as follows:

• Check each entry field that has maximum-length string data.
• Check each required field that has null data.
• Challenge type-checking validation of each field that has data of a different type from what is allowed.
• Test for all data values that are simultaneously maximized or most stressful.
• Verify that the nominal value of data is accepted, stored, retrieved, and displayed.
• Logon/logoff
• Password validation
• Open/close/maximize/minimize

In my next post, we’ll discuss how you can keep building and improving test automation to continue to streamline the process.

SOUND OFF: Why do you think test automation projects should be run just as any other software development project?

If the only goal of software testing is to improve product quality, then there is a lot of value being missed. Smart organizations realize that test assets developed during test effort are as valuable, or more valuable, if you consider the long-term benefit of having the right resources in place. Smarter organizations recognize that intellectual assets are the most valuable testing tools of all.

What seems to be overlooked is what impacts the creation of intellectual assets.

It is clear that the more one tests and the more things one tests and the more testing paradigms one operates in, the more intellectual assets one gains. However, in the economy of increasing testing-knowledge assets, the value of manual testing may be underrated and underutilized.

Why Would Exploratory Testing Be Underrated?

Knowledge comes from seeing and seeing comes from, well, from seeing. And what are the revelations the best tester never sees? Yep—what the automated test tool sees.

When a test is complex, especially if it is emulating user behavior, it seems sensible to execute some of the failed tests manually.  There will be lots of things for the wizened tester to see and think about when walking  through the steps that failed in the automated tests.

An automated test tool will never tell when it sees a missed test condition. On the other hand, manual testers are constantly calling out missed test conditions.

Automated Testing Does Not Remove The Need for Exploratory Testing

Overlapping automated tests with manual tests may not be cost effective, but replacing manual tests with automated tests may cost more in terms of product quality.

Do not stop manual testing just because automated test are in place and used. For quality (and your promotion’s) sake, keep on using exploratory testing.

My experience has been that exploratory testing finds the obscure, deadly bugs. The extra resources “wasted” doing seemingly duplicate exploratory testing is easier to take than the call that the production server crashed because a user dropped a book on their keyboard.

The best organizations will recognize the value of exploratory testing in all testing efforts. Not just because of the obscure bugs they reveal and the missed test conditions, but as importantly, what the testers see along the way.

SOUND OFF: In your testing organization, has exploratory testing ever revealed an obscure bug that automated testing missed?