The MITRE Engenuity ATT&CK Evaluations team announced the results of its most recent evaluation of enterprise security solutions, and I immediately started digging into the data for insights for security pros.
The MITRE Engenuity team is very careful to avoid any rankings or scores as part of the evaluation process, though some may see not providing rankings as a detriment and some participants may take advantage of it. To settle this debate, remember that MITRE ATT&CK emulates well-known threat groups inspired by publicly available threat intelligence. This is not product comparison. Detecting an attack inspired by a known threat actor should be the floor for what security products can do, not the ceiling.
MITRE Evaluations Are Not A Ranking, Which Is Why They’re Valuable
Every enterprise environment and security team has different needs, and the results of the evaluations tease out differentiation that can help security teams decide which tool to use. After a deep review of the data (which you can find in the full research and associated tool here), we found a few different ways security pros can look at these evaluations to help improve their detection strategy. The evaluations can and should be used to help you determine the following:
How Different Security Tools Detect
The crux of the evaluations is to show how different endpoint detection and response (EDR) tools detect to give practitioners insight into which tools may best support their needs. Security teams may want an EDR tool that gives visibility into all steps of the attack but does not detect them. By contrast, other security teams may want an EDR tool that detects every single MITRE ATT&CK technique.
A step-by-step view of what security tools detect becomes more valuable in the context of your specific environment. As an example, one of the techniques used in this evaluation is File and Directory Discovery. If an enterprise has an application, user, or set of users frequently traversing directories (which is most organizations), they will often see this technique used in a non-malicious way. Because of that, this technique may not be a priority for its security tool to detect on. In fact, it may prefer visibility into the technique (i.e., results were “telemetry” for the output) but not detect on it (i.e., results were “technique” for the output) or instead implement a preventative control for that technique.
Vendors that successfully detected on every single MITRE ATT&CK technique (i.e., results were “technique” for almost all or all outputs) may seem to “win” the evaluation on first glance. However, given that they are detecting on every technique, they have the potential to be noisy, have a high rate of false positives, and provide excessive alerting. Many vendors in the evaluation did not have visibility into common techniques, typically because these techniques are used for legitimate tasks in the enterprise. Contrastingly, vendors that had visibility into most techniques but did not alert on them (i.e., results were “telemetry” for many of the outputs) may have good visibility for threat hunting but could potentially have missed detections or require manual effort to create detections.
Where To Apply Compensating Controls To Build Resilience
Even as the evaluations showcase the vast coverage of participating security tools, they also show areas for improvement in security teams’ overall detection strategy. No single security tool can find and stop every single aspect of an attack on its own. By reviewing the results of the evaluation, security teams get an idea of where individual tools are most (or least!) effective. Use this information to identify gaps where it’s likely necessary to implement additional technologies like network, identity, or email security, among others.
How To Use MITRE ATT&CK Coverage To Move Toward Zero Trust
MITRE mitigations provide additional insight into preventative controls that can be implemented for many of the sub-steps in this evaluation to protect against this attack prior to detection by a security tool. We also align those mitigations to the Zero Trust eXtended (ZTX) ecosystem and NIST in our research to help security pros map potential protection improvements to their existing strategy.
Use The Evaluation As An Input Into Your Detection Strategy
The results serve as an input into a larger strategy discussion regarding the tools and processes the security team uses. Use the results — and the tool we created — to help your team understand how different security tools detect, where to apply compensating controls to build resilience, and how to use MITRE ATT&CK to move toward Zero Trust.
Last thing worth remembering: Failing to understand that these evaluations are not a ranking system will prevent you from truly understanding and getting value from the results. Don’t be that person!
A big thank-you to the MITRE Engenuity team for their work on the evaluations and for giving me their perspective for this report.
Read the full report here for more in-depth analysis and access to the MITRE Engenuity ATT&CK Evaluations tool.