Why Video Testing is better then Metrics Testing Alone
Until you have seen video testing, you may be skeptical about the power of what it can teach us about web site behaviors. By using the right approaches and asking for insights beyond basic (though also essential) task completion tests, you will learn what is and isn’t working on your site and why.
Video testing can be complicated and involved but it need not be. We know from independent research that tests of only 6 people will reveal over 90% of a site’s problems.
What this doesn’t say is that in addition to understanding the problems, careful testing can also reveal a range of solutions to these problems that simple or even complex traffic analysis tools or metrics testing, fail to reveal. By metrics testing, we mean the tools we use to analyze traffic logs which record visitor behaviors. These tools can measure statistics like page views, number of visitors, navigation paths thru sites, search keywords and orders and conversions. Metrics testing is helpful but we believe that video testing offers far more for a variety of reasons.
Video Testing is Quicker
We are promoting a new version of video testing which we call Rapid Deployment Testing. Larger companies may easily draw from a local pool of available candidates and run tests within hours of designing the test. During the design process, developers often come up with questions about best practices, in part because these practices continue to change and improve. What was great a year ago could well have improved by now. By testing one or two simple procedures, without the need for full scale site testing, in only a matter of hours we can gain actionable insights into these procedures. For example: What is the key messaging on a landing page? Is the call to action visible enough? Why are people missing a particular message?
Video Testing is Extendable
Smart companies will begin to build, and hopefully share in a “YouTube” environment, video clips of certain procedures so that stakeholders can review commonly tested behaviors like sign up forms or My Account shopping behaviors. Having these review materials readily available to designers will improve their design process, without the need to conduct new tests, thereby improving the quality of the prototypes and speeding up design. This assumes that a thorough video test will be run before launch to ensure that any unseen flaws will be revealed. Building a best practices library should reduce design questions, improve prototypes and, if it is updated, allow for the library to be continually improved as each stakeholder learns more and shares that learning.
Smaller Sites Can Benefit Immediately
Metrics testing is only helpful for sites if they have significant enough traffic to conduct a test. Typically, the longer a test takes, the less reliable it is. Ensuring valid statistical significance takes a deeper understanding of statistics then simply counting visits and conversions. Sites which lack the traffic to even obtain statistical significance are unable to use metrics as a reliable testing medium. Video testing gives smaller sites immediate results and input without the time needed to get the required traffic.
Video Testing is Impactful
We know that numbers are hard to argue with and that data can be compelling; however, Americans in particular are greatly swayed by video as well. If we see it on the small screen – we believe. Often when I present metrics analysis, the numbers seem to raise more questions, whereas video testing seems to raise more interest. Several short clips showing user frustration and annoyance go a long ways towards convincing reluctant stakeholders to make changes to a site. In fact, a peer ran a video test where she actually tested the company execs on their own site. The problems they exposed (the execs could barely use their own site), promptly resulted in seven figure funding for a full usability lab.
Video Testing Works as Well on Prototypes as it Does on Existing Sites
There simply is no metrics testing available for prototype sites. I don’t know how a site can be safely developed without testing early in the design process. Failing to test early and often has shown that we are doomed to costly design changes on the back end. By using the great prototyping tools for wireframing sites, such as Microsoft Visio and Axure RP to name two, you can build realistic prototypes which behave almost exactly like active sites. The agility of these design tools allows testing multiple graphic treatments and conversion paths of even relatively large sites in only hours. Video testing quickly allows us to measure the relative merits of each possibility.
Video Testing Measures Graphic Impact
It is very hard to measure how a site’s graphics will impact customers from metrics testing alone. In video testing, we simply ask about the look and feel of a site. If we understand how important images and graphics may be to a particular audience and determine that our audience has a higher percentage of visual learners for example, we will pay much more careful attention to visual impact. Metrics testing can compare which graphics may get more clicks, which is important. But why one works better normally cannot be determined by metrics, only face to face video testing can accomplish this.
Video Testing May Be More Accurate
I test in part because I know my limitations. I know from over 10 years of experience that people will not consistently react the way I expect. That they will not always respond the way I anticipate and they will not always understand what I want to communicate. Every time I decide that “this particular site can be launched as is”, I force myself to test, and always find significant room for improvement. (In all fairness, I do get it right a lot, but testing always makes my work better.) When I am using metrics testing alone however, understanding why people may click on one thing or another is usually based upon my best guess. Despite years of metrics testing and gaining certification in the skill, I am much more comfortable with the proof that video testing offers then the conjectures I am forced to make with metrics testing.
Video Testing is Reliable
Metrics testing can be reliable as well, however, there is always a significant margin of error in traffic reporting and we cannot predict if future browser changes or changing cookie behaviors will reduce its accuracy even over the short term. Video testing is quite indisputable. What you see is what you get. The biggest criticism is that the test sample is usually small. I agree, however the research on sample size is very clear. Even small samples turn up most problems (as I said over 90%).
Video Testing Gives Understanding on an Emotional Level
Buying is an emotional process. We think it is a calculated procedure, but it is much more emotionally based than most believe. Video testing gives immediate and honest understanding of the emotions of buying. No metrics tool I have seen to date gives me any insight into emotional behaviors.
Video Testing Provides Solutions
Metrics testing certainly shows that problems exist, but the only way to find solutions with metrics is to test a variety of solutions using a multi variable optimization tool, such as Google Optimizer or Optimost. These are good tools, however, they rarely actually reveal the problem in the way that video testing does. Clearly metrics testing can demonstrate that a call to action can be improved. However, video testing will immediately show when it is the message or the persuasiveness of the site that is the problem instead of the actual call to action. Metrics testing can also easily test multiple messages, however video testing can immediately identify the problem as being in the initial message and if the tester listens carefully and probes for understanding, you may be rewarded with a variety of new suggestions as well. This is one small example of the ability of video to offer solutions. It has helped in graphics, page elements, design, processes, and of course messaging.
Video Testing is a Great Pay Per Click Tool
Metrics has dominated the PPC arena while video testing has yet to make much impact there, and I don’t understand why. All our video testing of web sites begins with search; we measure the impact of search results and PPC ads on an emotional level. We watch to see what the behaviors are and why competitors may draw more clicks. In a recent test, we asked every participant if they ever clicked on PPC ads and 40% said: “Never.” That percentage may change in the future but it certainly led us to try to make changes to how our organic results appeared on Google. When we went on to test landing pages and more importantly the conversion path, what we learned will insure great returns for the client. Why is that so many PPC professionals seem to stop testing with the landing page? The entire conversion path needs to be tested and there is no better tool for immediate understanding than Morae and a good consultant.
Video Testing Isn’t THAT Expensive
It probably costs less than you think, though it is not cheap. Even starting from scratch you have most of the essential equipment except a web cam. You do not need a lab, in fact in many cases it can be preferable to test in other environs. You do need the software and the cost is under 1500 about the same as a good consultant costs for one day. You will have to pay your test candidates from $25 to $50 per hour, more in the big cities. Metrics testing is less expensive unless you choose to purchase the more expensive traffic analysis tools (which many experts feel are necessary). Google Analytics is free and getting better all the time but deeper keyword analysis still seems to require higher end programs. Whatever way you look at it, though, metrics testing is less expensive. However, if you run the simple cost justification tool Dr. Deborah Mayhew published in a recent TechSmith article , the cost of testing is negligible compare to the huge benefits it offers. Bottom line, total software and hardware cost for video testing, (assuming you have a computer) is under $2,000. It is hard to quantify what a consultant will charge for a video test, though the range of $3,000 to $50,000 for a full blown large scale site test is probably realistic. If we begin to amass a user experience library (hint hint TechSmith), we may all reduce some testing costs to zero, though we will never eliminate the need.
I realize that metrics testing offers many powerful insights and that is in an essential part of online marketing. In the interest of improving our “art”, I welcome its proponents to share their insights and refute or comment on my points. I look at traffic metrics every day, and I will continue to do diligent traffic analysis.
Video testing is certainly more complicated and challenging than metrics review, and this often translates to higher cost. But good basic video testing is within the purview of the small business person now. We gladly train our customers so that after we leave they can continue testing on their own. I should say that advanced user testing skills take time and learning to develop. However, don’t let that keep you from diving in.
To be honest, when we recommend a testing strategy going forward, it always includes keeping a careful eye on traffic metrics as well. As much as we would like to think that web marketing is a solid science, the truth is that one of the things that make it most exciting to me is that we learn more every day, and anything which expands that learning (or challenges it) is a good thing, be it metrics or video analysis. I suggest that you consider using Morae in new and creative ways, and I guarantee that you will find the results will improve your bottom line.
“Morae 201: Why Video Testing is better then Metrics Testing” first appeared in Techsmith’s: News you can use – usability newsletter.
For More Information