Hadoop Summit 2015

1. Hadoop Summit 2015
CloudFire Analytics: Transforming Security
Analyzing Symantec’s security data lake
Stephen Brodsky and Darrell Kienzle

2. Hadoop Summit 2015
Outline
CPE CloudFire: Analytics and Products1
Analytics Services and Data2
Analytics Administration and Monitoring3
Self-Service Analytics and Dynamic Clusters4
Symantec CloudFire Analytics 2

3. Hadoop Summit 2015
CPE CloudFire: Analytics and Products
• CPE – Cloud Platform Engineering
– Symantec’s Private Cloud for security products and analytics
– Spans 50+ data centers around the world
• CloudFire – CPE’s scalable cloud platform
– New data centers, new hardware, scalable build-out
– All open source
– OpenStack for virtualization
– Analytics for big data analysis
• Cloud for bringing together and integrating Symantec’s:
– Products
– Big Data, Analytic applications, and Services
– Compute, Network
Symantec CloudFire Analytics 3

4. Hadoop Summit 2015
Analytics for supporting Security Products
Goals of the Security Teams
1. Do we have the data?
2. Can we analyze the
data?
3. Can we provide timely
Insights?
CloudFire Analytics for supporting Security
1. Data available
• All frequently used Security Data available
• Data at scale: Data available for parallel analysis
(PB scale)
• Leveraging CPE Data Center Availability,
Compute, Net
2. Analysis engines
• Hadoop ecosystem engines (MR, Hive, Kafka,
Spark, HBase, Storm, Phoenix, ++)
• Analytics Pipeline
3. Analysis in near real-time
• Analytics timescale days/hours -> seconds
• Batch -> Streaming
4Symantec CloudFire Analytics

5. CPE Analytics Architecture Overview
Inbound Messaging
(Data import, Kafka)
Products
Distributed Storage (HDFS), Metal or Virtual (OpenStack) Servers
Analytic Applications, Workload Management (YARN)
Stream Processing
(Storm, Spark)
Real-time Results
(HBase, ElasticSearch)
Query
(Hive, Spark SQL)
Device
Agents
Telemetry, Data
Threats: Top Web-based
Attacks
192 web-based attacks recorded last month
36%
30%
14%
10%
7%3% Malvertisement
Exploit Kit
Suspicious
Download
Analytics Clusters
• All major open
source engines
• PB-scale Data Store
• Key Telemetry
• Multi-Data Center
• Administration
• Monitoring
• Prod, Dev/Test
• Application
Deployments
CloudFire Analytics
Data Transfer
5

6. Hadoop Summit 2015
A key security Product
Symantec CloudFire Analytics 6
• Live with hundreds of external customers
• Improved time to analyzed results from hours to seconds (5000x) at production
scale
• Leverage Streaming Analytics
• Move analytics to more powerful, high performance open source technologies
– Kafka for queuing
– Storm for analytics
• Improve analytics
– Server reputation
– URL reputation
• Moving forward
– Leverage Analytics Pipeline
– NoSQL DBs
– Graph DBs
HDFSMonitors

7. Hadoop Summit 2015
Analytics Services and Data
Symantec CloudFire Analytics 7

8. Hadoop Summit 2015
Product Journey with CloudFire Analytics
Symantec CloudFire Analytics 8
SDAP current
• Scale?
• Availability?
• Usage?
• Hours/Days
timeframe
Batch Analytics
•Data Scale: Multi-PB
HDFS
•Analytics scale: Hadoop
MR, Hive
•Availability – high
•Usage prioritization
•Minutes/Hours
timeframe
Streaming Analytics
•Analytics Scale: Kafka,
Storm, Spark,
ElasticSearch, HBase, ++
•Availability – high
•Usage prioritization
•Seconds timeframe
Application style
• Bring in data
• Analyze
• Architecture: Serial->Parallel
• Workflow oriented
Application style
• Bring in data in zip
files
• Unzip, load, SQL
• Architecture: Serial
• Application oriented
Application style
• Stream in data directly
• Analytics pipeline
• Architecture: Streaming
Parallel
• Streaming oriented

9. CloudFire Analytics Pipeline
Data Sources
Streaming
Analysis
Batch
Analysis
Actively
Available
Results
Online Access
Research
Telemetry Clients
World
Cloud
Data
Assemble Analytic Applications
Analytic Pipeline Message Queue
9
Storm, Kafka, Spark Hive, Spark
HBase, HDFS, Kafka, Cassandra
HBase, Index, Cache

10. Hadoop Summit 2015
CloudFire Analytics Services
Symantec CloudFire Analytics 10
Analytic Services and Data Transfer
• Hadoop, YARN - Compute
• HDFS/Knox/HttpFS – File Storage
• Hive, Pig – SQL Batch
• Oozie – Workflow and Scheduling
• Service Endpoints – Admin / BDSE
• Storm – Streaming
• Kafka - Messaging
• HBase – BigTable column store
• Spark (SQL,ML) – Stream analytics
• Research and admin CLI VMs
• MTS Parallel file Import / Export
• Falcon data management
• ElasticSearch indexing
• Cassandra, Graph, Drill, Solr
Web UIs
Admin Tools
• Ambari - Admin
• OpsView (Zabbix) – Uptime monitor
• YARN and HDFS UIs – Perf & Logs
• Ganglia & Nagios – Performance
• Storm & Kafka lag – Perf/Problems
• LMM – Problem determination & metrics
• Resource Manager, Name Node UIs
• Network data transfer monitoring
• Puma performance usage analysis
• Continuous Validation
• Ranger user management
Data Science Tools
• HUE – SQL, Jobs, Files, Workflow
• Ambari Views

11. Hadoop Summit 2015
Analytics Administration and Monitoring
Symantec CloudFire Analytics 11

12. Hadoop Summit 2015
Ambari Analytics Administration
12CPE CloudFire Analytics
Extensible administration platform for automated
deployment, monitoring and alerting, configuration
management, rolling upgrades, and rolling restarts.

13. Hadoop Summit 2015
Ambari + ElasticSearch – Ambari Stack extensibility
Symantec CloudFire Analytics 13
Ambari custom
service for
ElasticSearch, a new
service to enable
Ambari to manage.
Ambari deploy,
start/stop, config,
monitor
Uses the new
Ambari Views,
integrated via
iFrame
Elastic HQ admin UI
as our use case.
Keeps the
dashboard clean,
yet extensible.
On github to share…

14. Hadoop Summit 2015
Ambari Views with iFrame – Ambari extensibility
Symantec CloudFire Analytics 14

15. Hadoop Summit 2015
Cluster Usage Analysis using Analytics
Symantec CloudFire Analytics
15

16. Hadoop Summit 2015
Ambari Metrics
Rapid Metrics Dashboards Construction - Reliability
Symantec CloudFire Analytics 16
Prototype for
building out
flexible
dashboards of
most
interesting
application
and platform
metrics.
Alerting
available via
the LMM
service.

17. Hadoop Summit 2015
Ambari Metrics
Rapid Metrics Dashboards Construction - Availability
Symantec CloudFire Analytics 17

18. Hadoop Summit 2015
OpsView / Nagios monitor and alert service
Symantec CloudFire Analytics 18

19. Hadoop Summit 2015
Data Transfer Network Monitor
Symantec CloudFire Analytics 19

20. Hadoop Summit 2015
Data Transfer Rate Monitor
Symantec CloudFire Analytics 20

21. Hadoop Summit 2015
Self-Service Analytics and Dynamic Clusters
CloudBreak integration and Development In Progress
Symantec CloudFire Analytics 21

22. Hadoop Summit 2015
Self-Service Analytics and Dynamic Clusters
CloudBreak integration and Development In Progress
Symantec CloudFire Analytics 22
Purpose
● Create and connect Analytics Clusters:
● Authoritative Data Lake, Regional Hubs
● Development, Test, Integration, Staging, and Production
● Ability for CloudFire users to spin up new clusters to develop their applications.
Feature
● One-click spin up of Analytics clusters using OpenStack VMs
● VMs that developers spin up are part of their own OpenStack quota
● Developers will have admin access to the cluster they spin up, so they can
install more services if needed.
Automation
● We control the version of the software by using the same automation code
that we use for deploying our production clusters.

23. Hadoop Summit 2015
Creating and managing clusters
Symantec CloudFire Analytics 23
Cluster
Directory

24. Hadoop Summit 2015
Self-Service Analytics Architecture
Symantec CloudFire Analytics 24
Managed Dynamic Clusters
Cross-Cluster Capabilities
Cluster
Management
Cluster 1
Cluster
Directory
Continuous
Validation &
Monitoring
Cluster 2 Cluster ... Cluster N
Data
Catalog
Analytics
Tools
Each Cluster:
• Analytic Services
• Deployment
• Networking
• Monitoring integration
• OpenStack VMs +
Containers
• Deployed Applications
Assemble Analytic Applications
Analytic Pipeline Message Queue

25. Hadoop Summit 2015
Thank You!
Stephen_Brodsky@Symantec.com
Darrell_Kienzle@Symantec.com