At the end of the day, you need both capabilities. SIEM’s real-time correlation provides constant situational awareness; the Big Data principles can be leveraged to do the following:Perform tactical drill-down investigations in response to tactical alerts from situational awareness.Provide context to tactical processing.Build more intelligent tactical-correlation rules, based on conclusions from long-term BDSA.Troll wide and deep to identify ongoing attacks that are too low and slow to trigger SIEM alerts.
BDSA is turning out to be the next evolution of SIEM. Winning SIEM providers are ones who do the following:Embed technical innovations from the Big Data developer field Integrate with Big Data platforms for two-way flow of security intelligenceBuild advanced data-science methods into their correlation and analysis engines so that security analysts don’t need to be data scientistsEnhance data visualization capabilities to help humans recognize hidden patterns and relations in security data
Thanks to the schemaless architecture of NoSQL databases and the ability to store unstructured data, one of the promising aspects of Big Data is the ability to query across a broad swatch of different kinds of information (i.e., variety). But ironically, after going to significant effort to deploy a Big Data platform and feed it a variety of data, organizations can quickly find themselves building silos within the Big Data repository. Silos explicitly defeat one of the key value propositions of Big Data.
This challenge is what leads analysts to build silos within Big Data repositories. To make sense of data and ensure the veracity of the analysis, these analysts begin to define views that purposefully select data from a narrow swath of all available data. This silo phenomena is already manifest in some products positioned as Big Data. In perusing the solutions built on top of the platform, one finds a preponderance of applications that focus on machine data from a single technology (e.g., Microsoft Exchange), thus limiting the analysis to the perspective of that one application. If all you need is analysis limited to a single component of your network (i.e., a silo), a good supply of monitoring applications for Exchange and other server products already exists. Organizations that invest in Big Data must ensure that the project stays true to its mandate, or else the organization will simply be maintaining the same data silo in its Big Data repository that was once found in a point solution
No silos Dumping terabytes of information into a completely schemaless, unstructured database allows cross data-source keyword searching. But in section 4, "The Trap of Data Silos within Big Data Repositories," we pointed out that organizations run the risk of creating silos within the very repository that is supposed to deliver wider visibility. Security-event data is well understood after more than a decade of analysis by the designers at HP ArcSight. And such data is better served with a normalized event schema that identifies a given action such as logon failure as the same event across all platforms and log sources regardless of format. By normalizing all events into one common event taxonomy, ArcSight Connectors decouple analysis from vendor selection. This unique architecture is supported out of the box across hundreds of commercial products as well as legacy systems.
Slide Objective: Lay out the following key points narrative:Key Points:Market drivers, trends and opportunitiesBig security for big data: HP’s solution for big dataSecurity intelligenceSecurity analyticsContext based SIEMSemantic analytics and concept searchingRoI, proof points, etcQ&ATransition: so first, a look at what’s new and different in the landscape
99.5% of data is not tagged or analyzed. IDC predicts that 23% of the data is useful if tagged and analyzed. How much data are you analyzing today? Comprehensive monitoring and analysis is thus needed to extract value out of your dataSo how do you know if you have merely ‘a lot of information’ versus ‘Big Data’? If the information your organization is generating - or has access to but may or may not be capturing or analyzing
Imagine unifying the machine data across the IT in various formats from various vendors into a simple common format. With the unified tool you should be able to search for any information from any source without any domain expertise or through text-based searching. You can create reports, charts, and dashboards for compliance and regulations, perform quick forensic investigations or simply search through millions of events in seconds to quickly troubleshoot your IT.HP ArcSight Logger is a universal log management solution that unifies searching, reporting, alerting, and analysis across any type of enterprise log data making it unique in its ability to collect, analyze, and store massive amounts of data generated by modern networks. It supports multiple deployments such as an appliance, software, virtual machine, and within the cloud in both Windows® and Linux environment.HP’s approach to comprehensive log management solution is:Collect: Borderless collection of any data from any device in any format from 315+ distinct out-of-the-box loggenerating sourcesEnrich: While the data is being collected, filter and parse the data with rich metadata helping to unify the machine data across ITSearch: As the machine data is enriched during collection, you can simply search through millions of events in seconds on what you want through text-based keywords without any commands or domain expertiseStore: The unified data can be stored in any storage format that you have (NAS, DAS, SAN, etc) though high compression ratio of up to 10:1 eliminating the need for DBAs or expensive databasesAnalyze anything: the rich content built into Logger helps you to perform high-performance interactive searches, comprehensive drill-down reports, and real-time alerting to meet the needs of diverse teams to use machine data for IT Security, IT GRC, IT Operations, SIEM solution, and log analytics
Gartner in a recent report released in 2013 said that ArcSight has simplified the security intelligence and analytics through CORR engine.
Slide Objective: highlight the huge Time to Value improvements for a pair of real HP Customers.Key Points: HP’s Information Optimization solutions maximize Return on Information by accelerating Time to Value. With HP Autonomy, customers can analyze their unstructured (e.g. email, texts, video) and semi-structured (e.g. machine-generated) data. With HP Vertica, they can scale their structured data analysis to handle any dataset. When brought together they offer the only solution that bridges these two worlds. In addition, depending on the environment, HP can provide pre-packaged solutions in the form of Converged AppSystems solutions for SAP HANA and NetWeaver and Hadoop. The result is any customer can maximize top-line information value, minimize spend and optimize their Return on Information.
What is HAVEn: HAVEn is the #1 platform for big data in the industry.HAVEn stands for Hadoop Autonomy Vertica Enterprise Security and any n number of applicationsHAVEn is not a single product. It is a platform that consists of multiple components.As you see in the next slide we also have an HAVEn ecosystem around this platformHAVEn brings together everything you need to profit from big data; hardware, software and services. The 3 HAVEn platform components are connectors, applications, and engines.These are shipping already. We have 1000’s of customers using these components to build mission critical solutions.How does this all work together? As an example, one of the largest global banks does the followingWhen you call them, 3 things happen in parallel – your call gets logged into Hadoop for complianceYour call gets analyzed through autonomy for sentiment – to determine if the customer is happy or unhappy and this info is inserted into Vertica for real time analyticsSimultaneously, another thread gets other business info on this customer and merges it together to find if you are a profitable customerThis information along with other information is analyzed in Vertica in real time to determine how to effectively handle the customer. Should be be offered any promotion or discounts. Details on connectorsWe have 400 connectors from Autonomy and 300 from Arcsight that help you bring all kinds of data. With these many connectors, it is highly likely that you will be able to have off-the-shelf connector to your data.In addition each of the engine components (Autonomy, Vertica and Arcsight) also provide additional data connector frameworks and tools to help you write custom connectors .Additionally the HAVEn platform supports popular frameworks like Hadoop flume and Chukwa. And it is open to all ETL frameworks. Details on engines (For more details refer to individual product pages)Many HP customers use Hadoop or experimenting with it. HP believes in a open Hadoop strategy. HP has been shipping preconfigured Hadoop appliances and/or reference architectures with all major Hadoop vendors – Cloudera, Horton works and MapR. . What we are seeing is that Hadoop is great as a data store to bring in all kinds of data and for ETL, but customers are telling us that they want better engines. As an example Novartis switched from using Hadoop to Vertica and the processing went down from several hours to several seconds using Vertica. That meant rapid drug discovery. The impact saving livesAutonomy has the leading algorithms protected by tens of patents for human information processing - video, audio, text –ex in London Olympics, camera images captured in London were matched in real time to terrorist database. The impact – saving lives. It is one of a kind technology.Vertica is designed ground up in the last ten years in MIT. It was designed for the peta byte wave for blazing fast real time analytics on peta byte size sets. It is designed as analytics platform that supports standard SQL/JDBC/ODBC and R natively. But most importantly because it is designed for large data analytics you can do it at a fraction of what legacy systems cost. Arcsight has been the leader Security and Events Information mgmt. system on Gartner MQ for years. It is used by some of the largest organizations in the world. It has been proven to scale at a million events a second range. Details on applicationsWe have started modifying our existing application portfolio to use HAVEn. And we are building new applications that leverage power of HAVEnAs an example, HP has launched a new application for operation analytics which leverages the power of multiple HAVEn components.Many customers are already building applications that use multiple HAVEn components togetherTo help you get started we have lined up partners and SI’s that can help you build these solutions. Which brings us to the next point – the HAVEn ecosystem
Big Data Security Analytics (BDSA) with Randy Franklin
Top 5 Truths about Big Data
Hype and Security Intelligence