Mobile App Functional Testing, Device Labs, and Open Source
In the latter half of last year, I started researching mobile application testing tools. My research focused, so far, on functional testing, primarily around mobile app front-end testing. As I began the research, it became clear that the automation capabilities testers needed to validate app UIs was there, but application development and delivery teams felt that device labs were too expensive to be practical. During the research for the Vendor Landscape: Front-End Mobile Testing Tools report, we expected that device labs would be a differentiator among products only to discover that most of the major mobile testing solutions provide them in one way or another. There are differences between vendors when it comes to the flexibility, configurability, and management of their device lab offerings, but if you’re delivering customer-facing mobile apps you can do much of your testing on physical devices (our recommended method).
In earlier reports, we recommended that, because of the cost of on-device testing, development organizations focused their testing efforts on the most important aspect of their apps, letting users find issues in less popular areas of the app for them. With most of the major mobile testing vendors offering device labs plus Amazon and Google’s entry into the device cloud space, competition will drive down cost and make on-device testing the more common option for mobile app testing. Microsoft’s acquisition of Xamarin now gives Microsoft a robust and capable device lab, stuffed with a variety of Android and iOS devices, which adds to the competition in this space as well.
We recently published The Forrester Wave: Mobile Front-End Test Automation Tools, Q2 2016. In the Wave, we put ten testing vendors through their paces and ranked them accordingly. As expected, device lab capabilities is a differentiator between vendors as is the vendor’s focus on open source solutions. We noticed that vendors of proprietary solutions have started implementing support for open source testing frameworks; the most popular are Appium and Selenium, but other solutions are available as well. A few vendors are sticking with their proprietary solutions, but most have at least also added support for open source solutions. We expect that open source solutions will be the norm and use of proprietary solutions will dwindle over time.
We’re starting to see vendors add analytics capabilities to their testing solution. Not analytics from the standpoint of what users are doing with their apps, but analytics as a reporting option in the testing solution. Typical test results tell you what worked and didn’t work in your app; developer and tester efficiency improves when statistical data helps them understand the overall quality of their tests and test suites. Being able to identify flaky or unstable tests more easily will help streamline mobile app testing.