See why the 15 providers selected matter most and how they stack up in our 2018 Forrester Wave™. In our 26-criteria evaluation of omnichannel functional test automation (OFTA) providers, we identified the 15 most significant ones — Conformiq, Eggplant, Experitest, IBM, Micro Focus Silk, Micro Focus UFT, LogiGear, Microsoft, Parasoft, Perfecto, Ranorex, Sauce Labs, SmartBear Software, Tricentis, and Worksoft — and researched, analyzed, and scored them. This report shows how each provider measures up and helps application development and delivery (AD&D) and software QA professionals make the right choice when adopting Agile+DevOps and testing their features in continuous delivery pipelines.
This year’s Forrester Wave signals a firm change from my previous testing Wave evaluations, in which I segmented testing tools that addressed mobile apps mainly from those testing web/client server/desktop-based apps. The world is becoming a mobile-first one but has not yet become mobile-only. Will it ever? For this reason, the prior “The Forrester Wave™: Mobile Front-End Test Automation Tools, Q2 2016” and “The Forrester Wave™: Modern Application Functional Test Automation Tools, Q4 2016” are now merged into one. But what does mobile-first mean? For testing purposes, it means that developers think and design first for small mobile screens and devices, with fancy, new, and engaging GUIs, and then retrofit into the larger and heterogeneous world of desktops and laptops, which are still used for daily jobs by business users and often by consumers, too. Responsive web design helps adapt and keep the look-and-feel consistency of web browsing across all sorts of channels. In other words, testers have more to test and not less, and it’s quite heterogeneous, with a priority on mobile. In fact, enterprise DevOps pipelines have it all, and testing needs to be more comprehensive across both mobile and non-mobile but no less faster. Testing tool vendors need to adapt to the trend. To do that, besides omnichannel and other criteria, we evaluated:

  • Continuous delivery testing. Can functional test automation across multiple channels, including mobile, be integrated in fast CI/CD cycles easily and efficiently? Can functional test assets be easily leveraged for automated performance testing in the same pipeline, helping shift left for nonfunctional as well as functional?
  • Testing smarter, not just at scale. While scale and test execution speed matters, as well as the automated orchestration of tests, the real reward comes from automating design of tests. This is where smarter AI and ML comes in.
  • CX, multi-persona testing, and shift-left support. We paid attention to criteria that indicates vendors are “at least starting” to extend testing to customer experience, criteria that evaluates multiple personas (including developers and business), and criteria based on shifting testing further left and right by leveraging production data/information.

The decision to include these different vendors follows a precise rational: that of a market in transition. We tried hard to select those that we think have the potential to lead the market, but using our rigorous relative comparative Forrester Wave methodology, we also tried differentiating the Leaders from Strong Performers, Contenders, and Challengers. There is some consolidation happening in the testing tool market, but it’s far from making it a static market just yet. Reach out to me if you would like to discuss the Forrester Wave evaluation results further.