Big Data Security Tools

On the May 30th Virtualization Security Podcast, Michael Webster (@vcdxnz001) joined us Live from HP Discover to discuss what we found at the show and other similar tools around the industry. The big data security news was a loosely coupled product named HAVEn which is derived from several products: Hadoop, Autonomy, Vertica, Enterprise Security, and any number of Apps. HAVEn’s main goal is to provide a platform on top of which HP and others can produce big data applications using Autonomy for unstructured data, Vertica for structured data, Enterprise Security for data governance and hadoop. HP has already built several security tools upon HAVEn, and I expect more. Even so, HAVEn is not the only tools to provide this functionality, but it may be the only one to include data governance in from the beginning.
There are several products that couple big data analytics with the goal of answering tough security questions as quickly as possible, with the toughest being “Was I hacked?” But why is this difficult and why do I need big data tools such as hadoop to help solve this? Consider the nature of data used for security, it is comprised from:

• System Log files that, during an error, can grow by gigabytes per second
• Network data captures, which can grow by 10s of gigabytes per second, which is one reason these are used sparingly
• Intrusion Detection/Protection log files, which can show just the headers or even the packets within network traffic which can grow by 10s of gigabytes per second
• Application Log files that, during an error, can grow by gigabytes per second
• Application Performance Metrics that have a resolution of seconds and imply an immense amount of data

All this data can turn into terabytes per hour. I do not know about you, but I cannot investigate terabytes per hour without some help. Into this breach come the big data security tools that are detecting attacks sooner than before. While this is not the only use for HP HAVEn, it was a demonstrated one as HAVEn unlike other tools for big data security analysis is a platform on which to build tools. Here is a list of possible tools to use, there are others that I expect to be discussed at the Misti Big Data Security Conference.

HP HAVEn is a platform on which to build tools that includes HP ArcSight and the use of its logger to pull in machine data (logs) while providing the ability to handle data governance of the data pulled into the big data repository whether that is Autonomy, Vertica, or Hadoop. However, this is on ingest or access to the data, not on output of any derived content.
RSA SilverTail started as specifically design tool to ingest web log data and to find fraud but has expanded to look at nearly all machine data (logs) as well as networking data to specifically to perform behavioral analysis looking for those unknown unknowns that could point to security issues.
Splunk App for Enterprise Security has its roots in Splunk Apps used for performance management of an environment. Splunk has gone a step further to analyze this data plus machine data (logs) to determine if the behavior of an application has changed in such a way to represent a security concern. They are looking at url as they go into and out of an application for example as well as other items that could point to an unknown unknown.

These three big data analytics applications have been designed for use by security teams to find out those unknown unknown security issues that stem from looking at behavioral analysis of an application. Behavioral analysis of people and applications is the current trend in security analytics as it is nearly impossible to prevent attacks unless you can recognize them, and recognizing attacks is becoming more difficult unless we switch to using analytics to see across a wide range of non-uniform data.
As an aside, we also discussed on the podcast, a few steps for security while at conferences. More on that next time.
I would like to thank VMware for the loan of their meeting space at HP Discover and HP for inviting and paying for my trip to HP Discover.