Today, AVG announced the acquisition of Sana Security, a longtime host-intrusion prevention software vendor. I have particular affection for Sana because they were a former client of mine at a previous job. Back in the summer of 2007, when security startup venture money was still flowing freely, like a rose-scented fountain at a Vegas casino, I remember giving a speech for Sana at their San Jose Grand Prix event. Don Listwin, their then-CEO, was a serious car racing enthusiast. He had conspired with the city of San Jose to shut down the city center so they could run race cars down the middle of it. It was pretty wild stuff — speaking as someone who comes from Boston, where all of the roads seem to be derived from old horse-trails or giant spiderweb patterns.

Host intrusion prevention software has always been a fascinating subsegment of client security, not least because of the fact that what HIPS vendors try to do is actually pretty hard stuff. In concept, the idea sounds simple: monitor processes in memory for suspicious activity, and block them when they try to do something naughty. For example, an ActiveX control executing in the context of a website should not be allowed to open a command shell and then initiate an outbound connection to somewhere else. Simple, right?

In practice, though HIPS isn’t so simple. Some early vendors — like Cisco’s CSA, née Okena; or Entercept, acquired by McAfee — relied on rules to enumerate behaviors that would be allowed or blocked. That worked, but only after lots of tuning. And re-tuning. And even more re-tuning. Anybody who’s every written declarative security policies (firewall rules, Java security policies) knows how tough this is to get right from an engineering perspective — the rule language needs to be precise, but flexible at the same time. These early generations — since improved — went through significant growing pains to be useful. (Hmm: activity monitoring driven by rules that require lots of tuning… reminds me of DLP today!)

Second generation HIPS vendors, like Sana and Prevx, relied less on rules and more on something akin to fuzzy logic. An activity received weighted scores based on the context of the process, what it’s doing, its code “pedigree” and other factors. Activities that seemed “suspicious enough” got blocked. This cut down on the need to create lots of rules, and made second-generation products less prone to get in users’ faces. Sana, in particular, sought to make their tool relatively “noiseless” — an important consideration in the consumer market.

Getting the technology right was hard enough. But recall also that malware authors have become much more professional and clever in their techniques. The increased sophistication of attackers is a fact of life: see, the Metasploit project, or the Uninformed journal for some examples. Today, it is an arms race. HIPS program want to burrow as deep into the Windows kernel as possible, to provide the best protection. Malware wants to burrow even deeper to avoid detection! Users therefore face the prospect of security programs and malware both fighting each other to get the deepest into the kernel, sort of like two dogs digging for a bone. Of course, kernel hooking falls entirely into the realm of undocumented voodoo, so instability is often the net result of this sort of alpha-dog antics. This was precisely the reason why Microsoft shut down (or tried to) access to the kernel in 64-bit versions of Vista.

As an analyst, Sana was always a hard company for me to evaluate. Most of their customers are consumers, so it is difficult to get credible information about how well the product actually worked. That said, the head of a large anti-virus company’s engineering group (not AVG) told me about a year ago that they admired their technology very much. But they haven’t been generating any buzz for quite a while. Their website is badly out of date. Sana has been shopped around for over a year, drawing interest from the Usual Suspects in Silicon Valley. I suspect the prospective acquirers I heard about last year concluded that Sana’s HIPS features weren’t tons better than what they already had, or at least no so dramatically better that they would justify the price of the acquisition and follow-on integration efforts.

With this deal, I think we can safely declare the era of stand-alone HIPS to be over. HIPS functionality is pretty well integrated into mainstream security suites now, as I and many other analysts have predicted it would be. Symantec and McAfee bought their own HIPS products several years ago, and have moved HIPS features into the core. Panda and Sophos have developed their own indigenous HIPS features. This is a good thing, of course — but it shows how, in retrospect, HIPS was always meant to be a “feature” and not a “product.”

Farewell, HIPS. It’s been fun.