Big Data: Enabling Us to Discover the Right Questions

For years I have been pondering how Big Data can help our challenges in information security. Let me start with a few clarifications about what the term Big Data means.

I think of Big Data as very large data sets that are housed in some sort of cloud-based data analysis repositories. However, unlike traditional data archives that sit and collect virtual dust on a few archive server 'thrones', Big Data repositories employ a vast farm of server 'folding chairs' that each does their small part to provide next-generation database analysis and mining tools. It’s democratization of data access and it makes mining and discovering valuable insights from the data both economical and possible.

In our world as a managed security service provider (MSSP), information security is mostly about collecting the data in logs and monitoring it. But traditionally when we think about monitoring data from a security perspective, we care mostly about detecting the bad things that are happening, then stopping those bad things from getting worse. This negative security model detects countless attacks every day but has its limitations. Today the million dollar question is, "How do we know when something 'not good' is happening?" The benefits of combining a negative security model where we are looking for 'known bad' and a positive security model where we know what 'good' looks like and can look for 'not good' are enormous.

It enables us to detect attacks we have never seen before by focusing on anomalies from some validated baseline, while still being able to continuously build better rules and signatures for 'known bad'. That is where, I believe, Big Data shines.

Within the Solutionary ActiveGuard® service platform we employ dozens of analyzers that know what to look for and can tell us when something either 'known bad' or 'not good' may, in fact, be happening. The work involved in developing and maintaining these analyzers and the rules they use are difficult, and far from trivial. The devices we take log feeds from are constantly changing and evolving, as are the types of threatening activities that our analyzers are designed to identify. That makes our work more interesting, but also more challenging over time.

As our customers recognize the value in log monitoring, they ask us to monitor more and more diverse aspects of their infrastructures — from mainframes to wireless endpoints, to virtual infrastructures, to enterprise custom applications and more. Adding more and more device types increases the diversity of logs we must understand how to monitor and the 'good' baselines we need to validate.

This is where Big Data is helping. The ease of mining log data using big data technologies at our research labs continues to augment the traditional research means we employ to discover the patterns that drive intelligent analyzer and rule design. Big Data is helping us discover new attack patterns faster than before, and with its help, are comforted knowing that our analytics are staying at the bleeding edge and that our customers are safer than ever.

See how Solutionary managed security services based on the patented ActiveGuard service platform combine security intelligence and expertise to provide visibility, threat detection and event response.