Last week, I attended the AV-Comparatives conference in Innsbruck, Austria. This conference brought together many cybersecurity vendors, particularly those with a European focus, as well as a few nonprofits, academic institutions, and analyst firms. The event was a combination of talks and an award ceremony, a typical arrangement for a conference from a security testing lab vendor.

Conference organizers asked me to give a talk on how we as analysts use third-party lab testing results like those from AV-Comparatives, MITRE Engenuity, SE Labs, and others. It provided a good opportunity to give a look behind the curtain as to how we use these results in our conversations with clients.

My predecessor, Josh Zelonis, was the first Forrester analyst to include the MITRE ATT&CK evaluations as a requirement for inclusion into The Forrester Wave™: Endpoint Detection And Response Providers, Q2 2022. I continued that trend for the past several years. The reason? As comprehensive as the Forrester Wave is — and you can see just how on our methodology page — we don’t test the technology in a lab.

We use three inputs for the Forrester Wave: 1) the questionnaire, which is typically several questions per criteria about the product; 2) the strategy briefing and demonstration, so we can see the product in action and learn more about the vendor strategy; and 3) the customer interviews, where we hear directly from customers about their relationship with the vendor. This gives us a great picture for an executive-level audience as to exactly what a partnership with the vendor will look like. It lets us get direct feedback from existing customers and identify the strengths and weaknesses of different offerings.

Third-party lab tests give us input into how effective the tool is (or isn’t) against specific attacks. To be clear, we do not use third-party lab tests as an input into the Wave, but we do use them to get a more complete view of vendor capabilities. I use these results to do three things:

  1. Validate how effective the vendor is for a particular attack scenario and where potential gaps are.
  2. Interpret the results from the entire cohort of results to get a better sense of how the technology market is changing.
  3. Help push the industry forward by highlighting gaps in the products or ways it may not be as effective for customers.

I want to ensure that we have a complete picture of the technologies we cover, which is why I review third-party lab results for my security operations research. It’s one more input into the bigger picture of how effective the vendor will be for each client’s specific use case.

We are also careful to evaluate the third-party lab tests we consider, ensuring that they are not pay-to-play, that they prevent the introduction of potential biases, and that all vendors are treated equally as part of the process. We never parrot third-party lab test outputs. Instead, we focus on our own analysis of the data.

If you have more questions about how we use third-party lab results, or if you want to talk about a particular vendor, book an inquiry or guidance session with me.

Lastly, Innsbruck is a beautiful town — if you have a chance to visit, definitely do so. And make sure you bring your ski gear!