In the past 18 months I've spent a lot of time working with Forrester clients on many facets of online testing (that’s a/b and multivariate testing for those of you scoring at home) spanning vendor selection, organizing and developing skills for testing, and building processes to support testing.
One of the general trends in online testing has been the democratization of access to marketing users. I think this is a positive development because successful online testing is a team sport that requires collaboration across multiple departments and skillsets. However, pulling testing outside of the exclusive domain of analysts puts a lot of pressure on vendors to supply tools that are suitable for non-technical audiences. This means providing easy-to-use, guided functionality, collaboration features, campaign preview facilities, extensive object reuse, and modern interface designs. And, to varying degrees, vendors are making progress in the area of user experience to meet these needs.
I have noticed that one of the features that often gets short shrift is test planning tools. In my experience, planning functionality has come forward as a crucial – and underrated – feature in situations where marketers or non-technical users will be involved in the development and deployment of online testing campaigns. To explore this idea further, I just published a new piece of research titled "How CI Professionals Can Plan For Site Optimization Success."
An online testing planning tool is essentially a calculator designed to evaluate the parameters of an experiment to determine the expected duration of a test. Typically, the functionality is available natively within the online testing application, but occasionally may be available as a standalone application from the vendor or as a homegrown macro or spreadsheet. Planning tools provide visibility to help users tune their experiments to complete within an appropriate time window or for specific situations; for example, varying amounts of site traffic (perhaps seasonality or different web properties) or different conversion rates for different types of promotions.
As you look at online testing tools, it is crucial to understand the day-to-day users of the application. Organizations planning to deploy an online testing tool to non-technical or marketing users must take special care to evaluate application usability – including the availability of planning functionality – to ensure that those users will be productive in the testing environment. Poor usability will become an obstacle to overall application adoption and lack of planning tools may impede appropriate test design, both of which can cripple an organization’s ability to scale up a site optimization program.