Blog

Written in collaboration with Travis Moser, Experience Architect

Setting up Google Analytics to capture site metrics is usually a pretty straightforward task. However, sometimes it requires a bit of creative thinking when you run into unexpected challenges.

Graphic by David Vincente. Some rights reserved.

The Challenge

On a recent project, we were installing Google Analytics to capture data for a website that was contained within a sub-directory of a larger corporate site. The business owner wanted to capture data specific to her sub-site—in particular—how many visits to the sub-site it took for a user to submit a contact form. It was important for the business owner to know this to make a judgment on how well the content was doing to explain the offering and persuade the user to take the next step. A reasonable request, right?

Here’s where the challenge came in.  Because marketing campaigns were going to be sending visitors directly to the sub-site, the business owner needed to record visitors to the sub-site separately from visitors to the larger corporate site.  Currently, the cookie tracking unique visits was being written to the corporate domain. So, for example, if a user visited the corporate site two times, then visited the sub-site twice and submitted a contact form on the second visit, Google Analytics would show that it took this user four visits to submit a contact form. We would have no way of knowing how many of those visits prior to the last visit were actually visits to the sub-site.

We called on Experience Architect Travis Moser to do a little investigating.  Here’s what he learned and how he recommended resolving the problem.

The Solution

We needed to figure out how to show that even if visitors had previously visited the larger corporate site, they should be counted as new visitors when they arrived on the sub-site. At the same time, we needed to track these visits to the sub-site to the larger corporate site Google Analytics account. Essentially, we needed to track the visitors to the sub-site twice—once to the larger corporate Google Analytics account—and once to the sub-site Google Analytics account, while making sure the data for one account didn’t affect the data for the other.

On the surface, that doesn’t seem too difficult. Let’s say the domain name for the site is http://www.acme.com and the sub-site is at http://www.acme.com/sub-site/. After some research, this is the code that we found others using:

var _gaq = _gaq || [];
gaq.push(
  ['setAccount', 'UA-XXXXA-X'], // global site account number
  ['trackPageview'],
  ['ss.setAccount', 'UA-XXXXB-X'], // sub-site account number
  ['ss.trackPageview']
);

The problem with this code is that because both site accounts use the same domain name, their cookies will conflict, resulting in inaccurate visitor counts. The accuracy of visitor counts was critical to determining the effectiveness of the site content. We realized we needed to force Google Analytics to set two separate cookies—one for each account. This can be accomplished by either setting a separate domain name for each account or by setting the cookie path for one of the accounts to a different directory. Because the sub-site is in a separate directory, it made more sense to set the cookie path for the sub-site account like this:

 var _gaq = _gaq || [];
gaq.push(
  ['setAccount', 'UA-XXXXA-X'], // global site account number
  ['trackPageview'],
  ['ss.setAccount', 'UA-XXXXB-X'], // sub-site account number
  ['ss.setCookiePath', '/sub-site'], // different cookie for sub-site
  ['ss.trackPageview']
);

Now the data is tracked into two separate Google Analytics accounts with two separate cookies: the larger corporate site account’s cookie domain is set to www.acme.com, and the sub-site’s cookie domain is set to www.acme.com/sub-site/.  Problem solved! The business owner will have an understanding of how the new site content is doing to get visitors to take action, and the corporate site owners will still be able to accurately track unique visitors to the site overall.

Travis, you rock!

Follow the Rules of Software Development

Previously I spoke of the problems in attaining a level of success when implementing automated testing within a project.

For example:

• Spare time test automation (team members consumed by larger tasks)
• Lack of clear goals (is implementation of successful  automation processes the long-term goal or simply a wishful thought?)
• Lack of experience (inexperienced staff who may create short-term fixes without long term success)
• High team member turnover (when test automation is not its own project and team members better utilized elsewhere, or they simply run out of time to accomplish the automation goals)
• Reaction to desperation (struggling to learn and meet timelines simultaneously)
• Reluctance to think about testing (focus on automation prized above the testing itself)
• Technology focus (do the results meet the testing needs?)
• Labor intensive (management of the automation processes may prove to be far more extensive than manual testing if not created and managed as reusable resources)

This leads me to the conclusion that Test Automation should be looked at as a Software Development Project.

An organization is often oblivious to the fact that it is actually developing software; there is no distinction between the users and the developers. This is the place that test automation often finds itself.  Thus, dedicating resources to test automation and treating it like a development activity elevates it to the first level. This is the core of the solution to the problems of test automation. Test automation projects should be run just as any other software development project.

Like other software development projects, developers should be dedicated to developing the test automation.

• Test automation automates a task in which the developer is probably not an expert. Therefore, skilled testers should be consulted and should provide the requirements.
• Test automation benefits if the approach is designed before coding is started.
• Test automation code needs to be tracked and secured. Therefore, source code management is a requirement.
• Test automation will have bugs. Therefore, it is a necessity to have to plan to track them, test for them and provide change management.
• Users will need to know how to use the tests and the tools. Consequently, create and update user documentation.

Assuming that you are part of a software organization that already has some idea as to what reasonable and effective methods should be used for developing software; you are urged to abide by those rules established for software development in your own test automation. The goal is to maintain the concepts used for software development projects, alongside the considerations and challenges that are particular to test automation.

Examples:

1. Improve the testing process
2. Define requirements
3. Prove the concept
4. Encourage product testability
5. Design for sustainability
6. Plan for deployment

Intro to Automated Testing 101 (avoid initial high-end costs, training, and time investments)

As stated above, some of the most immediate goals of automated testing are as follows:

1. Improve speed and efficiency
2. Establish reliability of process
3. Create repeatable processes
4. Manage reusable processes

Additional objectives are:

1. Reduce costs
2. Prove the concepts
3. Improve the overall testing process

What efforts can be most quickly served through automation and positively affect the items listed above?

There at least two (2) primary conditions where the introduction of automation can create immediate cost and time savings with reduced training and lower initial financial investment.

1. Changes of environment, testing across multiple browsers (IE, Chrome, Firefox, Safari, etc.) or multiple browser versions (IE7, IE8, etc.).
2. Testing of systems that process large volumes of data i.e., retail, medical, research, etc.

In both of these cases, the tests remain essentially the same through each repetition. They can be created by recording keystrokes and mapping them to a script or by loading diverse data sets to test the flexibility of the system.

Test engineers start with a blank page in a test-design specification and fill it in detail by detail, when many of these issues are generic and common to nearly every application-testing scenario. Again, the introduction of recordable testing tools to create an initial package of test designs without programming and with minimal training is possible. These designs can be readily modified with the edit tools provided by relatively inexpensive testing suites (5 figures versus 6 to 7 figures) to adjust designs quickly for reuse.

Examples as follows:

• Check each entry field that has maximum-length string data.
• Check each required field that has null data.
• Challenge type-checking validation of each field that has data of a different type from what is allowed.
• Test for all data values that are simultaneously maximized or most stressful.
• Verify that the nominal value of data is accepted, stored, retrieved, and displayed.
• Logon/logoff
• Password validation
• Open/close/maximize/minimize

In my next post, we’ll discuss how you can keep building and improving test automation to continue to streamline the process.

SOUND OFF: Why do you think test automation projects should be run just as any other software development project?

If the only goal of software testing is to improve product quality, then there is a lot of value being missed. Smart organizations realize that test assets developed during test effort are as valuable, or more valuable, if you consider the long-term benefit of having the right resources in place. Smarter organizations recognize that intellectual assets are the most valuable testing tools of all.

What seems to be overlooked is what impacts the creation of intellectual assets.

It is clear that the more one tests and the more things one tests and the more testing paradigms one operates in, the more intellectual assets one gains. However, in the economy of increasing testing-knowledge assets, the value of manual testing may be underrated and underutilized.

Why Would Exploratory Testing Be Underrated?

Knowledge comes from seeing and seeing comes from, well, from seeing. And what are the revelations the best tester never sees? Yep—what the automated test tool sees.

When a test is complex, especially if it is emulating user behavior, it seems sensible to execute some of the failed tests manually.  There will be lots of things for the wizened tester to see and think about when walking  through the steps that failed in the automated tests.

An automated test tool will never tell when it sees a missed test condition. On the other hand, manual testers are constantly calling out missed test conditions.

Automated Testing Does Not Remove The Need for Exploratory Testing

Overlapping automated tests with manual tests may not be cost effective, but replacing manual tests with automated tests may cost more in terms of product quality.

Do not stop manual testing just because automated test are in place and used. For quality (and your promotion’s) sake, keep on using exploratory testing.

My experience has been that exploratory testing finds the obscure, deadly bugs. The extra resources “wasted” doing seemingly duplicate exploratory testing is easier to take than the call that the production server crashed because a user dropped a book on their keyboard.

The best organizations will recognize the value of exploratory testing in all testing efforts. Not just because of the obscure bugs they reveal and the missed test conditions, but as importantly, what the testers see along the way.

SOUND OFF: In your testing organization, has exploratory testing ever revealed an obscure bug that automated testing missed?

Our story begins with a client who needs your services. In this story, we have a business that is very profitable, and now this business has decided that they want to move to a software solution that will maintain their vitality and propel them into the future. As an additional note, the company does not have an in-house software development or testing team.

They hire you to help them with their testing. From the initial analysis gathered in a meeting with them, you recommend a risk-based approach to testing. The major reason for selecting this approach is that they have not budgeted enough time for the testing efforts. One guesses they did not budget enough time partly because they are new to software development and software testing specifically. So, the project is nearly complete with just a couple of months to go, and they still have not started testing. Conclusion: a risk-based approach is our best choice.

Let’s expand the picture to say the solution is huge: three different vendors are bringing in their software packages and they have to integrate so that total solution provides a seamless communication of both vital and non-vital information, as well as some very technical information, necessary to propel this  business into the future growth it desires.

In other words, you come in and see Goliath and know that a risk-based approach will best address this very challenging task: provide a measure of testing and an understanding of the software quality in an incredibly small time frame, considering the size of the solution.

More detailed reasoning for choosing risk-based testing

From our experience, risk-based testing is one of the best methods of documenting a testing approach while cutting costs and staying  extremely agile, which is good for a company just beginning to create testing processes and practices. It is less expensive and it has a faster cycle.  Also, we have found risk-based testing a great way to start if you want higher confidence that you are testing the right things at the right time.

We use the client’s  SMEs (subject matter experts) to define risk areas and then rate them so that we have information to drive the plan. Since you know which areas are critical, you start with the highest risk areas and work your way through the others. As part of this process, you want the project management team to create guidelines for success and completion. For example, the cycles of testing will be complete when you have tested all the high-risk areas three times; the medium-risk areas twice; and have done some analysis on the low-risk areas, based on the testing of the other two areas.

Here again, the time constraints are the driving factor (as well as some other context-driven situations, e.g. lack of written requirements). So we recommended and went with SBTM – Session Based Testing Methodology. This framework allows for a maximum “hands on” testing and a minimum of documentation and management of the process. Based on limited time, the need to maximize testing efforts and minimize documentation and other administrative efforts, we decided this was our best framework for testing.

We often hear “the devil is in the details,” but for us, in this case, the devil was getting to the details, which we just described as learning how to help the company move into testing and do it with the constraints that were given. Our solution was risk-based testing with SBTM.

As for the details, we worked within the time frame allotted and accomplished what we needed to, thanks to our ability to pick the solution that best fit our context. We did lots of testing; covered the areas that were expected to cover; and delivered the needed work and documentation.

The biggest lesson we learned from implementing risk-based testing using a Session Based Testing framework was “never stop asking questions.”

This is a story that does not end here, but continues every day. We like stories.

Some Useful Links
James Bach – http://www.satisfice.com/
Michael Bolton – http://www.developsense.com/
Mike Kelly – http://michaeldkelly.com/

SOUND OFF: Do you have a testing story to share? We’d love to hear it!

Testing is the last thing you do before delivering a digital project.  No need to stress about it up front.  Just include time for it in the plan, and we’ll nail down the details when we come to it. We’ll leave it to the experts to tell us what we need to do when the time comes.  Sound like a plan?

Yeah—sounded like a good plan to me until I was asked to do the testing.   Yikes!  I manage digital projects and develop content.  I consider myself more of a “creative,” for gosh sakes. You want me to test software? Gulp!

So, with some expert guidance, test software I did.  I became familiar with the language and format of test scripts. I executed them. I even graduated to writing a few. I found bugs and documented them.  I made sure requirements were met.  Interestingly, this experience, which I thought would be quite boring and mechanical, fed my inner perfectionism.

Formal testing was not something I’d ever been asked to do before or ever imagined doing.  I’m quite sure it’s not in my job description.  However, having been called on to do testing , not once, but several times,  has given me an appreciation for the skill involved, for those who are experts at it, as well as for its importance and value to every digital project.

The most important things I’ve learned from my experiences as a tester that have helped me as a project manager:

• Clearly define how and what you are going to test based on client needs and how the final product will be used
• Include adequate time and resources for testing in your plan and estimate
• Line up testing resources (people, hardware and software) early
• Document meticulously

I know there’s a lot more to testing than I’ve experienced so far.  That’s why we have an entire discipline dedicated to testing and why I’m excited to dig in with the members of our testing team to share their approaches to and perspectives on testing and to show you what makes them passionate about their work. And— of course—why I’d rather have them lead and do the testing  on my projects!

Stay tuned for my upcoming posts this month for testers’ top tips, how to work best with a tester, tester’s pet peeves and more.

Sound like a plan?