How to fight cyber security threats using Big Data tools - An introduction to Apache Spot (incubating)

1. 1© Cloudera, Inc. All rights reserved.
A Community Approach to
Fighting Cyber Threats -
Apache Spot (incubating)
Mark Grover | @mark_grover
Apache Spot (incubating) committer and PPMC member
Slides at slideshare.com/markgrover

2. 2© Cloudera, Inc. All rights reserved.
About the book
• @hadooparchbook
• hadooparchitecturebook.com
• github.com/hadooparchitecturebook
• slideshare.com/hadooparchbook

3. 3© Cloudera, Inc. All rights reserved.
Agenda
• Apache Spot (incubating)

4. 4© Cloudera, Inc. All rights reserved.
… while security operations
centers do not.
SOC 2
SOC 1
SOC 3
SOC 4
Hackers collaborate
everyday…

5. 5© Cloudera, Inc. All rights reserved.
Apache Spot
(Incubating)
A community approach to fighting cyber threats.
spot.incubator.apache.org

6. 6© Cloudera, Inc. All rights reserved.
Gaps in existing cyber security solutions
Detecting Advanced
Threats
Only signature and correlation
based detection
Machine learning is difficult to
implement
Data is not enriched for better
detection
Reducing Investigation and
Response Time
Access multiple applications in order
to act
Partial enterprise visibility
Only access months worth of data
Understanding True
Business Risk
Balancing risk with costs
Getting an understanding of the
risk of an entity or user
Meeting changing compliance
regulations

7. 7© Cloudera, Inc. All rights reserved.
Why Spot? Why now?
• Big data tools
• Scalable storage and computer
• Reasonable cost
• Commodity hardware
• Advanced threat detection
• Machine Learning

8. 8© Cloudera, Inc. All rights reserved.
The Value of Apache Spot
Detect advanced threats faster
via machine learning
Faster time to incident
investigation and response with
comprehensive enterprise
visibility
Change the economics of
cybersecurity with an open
source platform that supports
multiple LOB workloads

9. 9© Cloudera, Inc. All rights reserved.
Architecture diagram

10. 10© Cloudera, Inc. All rights reserved.

11. 11© Cloudera, Inc. All rights reserved.
Apache Spot Ingestion
Partnering with:

12. 12© Cloudera, Inc. All rights reserved.
Apache Spot Processing
Analyst
queries
(UI)
Analyst
full-text
search (UI)
Machine
Learning

13. 13© Cloudera, Inc. All rights reserved.
Remember Netflix prize?

14. 14© Cloudera, Inc. All rights reserved.
What if…
• … we combined netflow, DNS, proxy data with
• User context
• Org, privileges, etc.
• Endpoint context
• What security regulation governs this server
• Network context
• Information about network from whois servers, etc.
• Threat intelligence model
• Set of known malicious IPs, etc.

15. 15© Cloudera, Inc. All rights reserved.
Open Data Model
• Raw event
1Zg2y780a,10.1.1.3:23444,10.1.1.10:1521,successful login as sysdba by jsmith, Oracle
• User context
John Smith, jsmith,smithj,csdkkv,jsmith@companyA.com,Jeff Beck,703-555-1212,
Recruiter, domain users, HR
• Endpoint context
10.1.1.10,crm.companyA.com,IT,Prod,SOX,PCI,Redhat 6.1, Oracle CM,
jt@companyA.com

16. 16© Cloudera, Inc. All rights reserved.
Open Data Model
• Raw event
1Zg2y780a,10.1.1.3:23444,10.1.1.10:1521,successful login as sysdba by jsmith, Oracle
• User context
John Smith, jsmith,smithj,csdkkv,jsmith@companyA.com,Jeff Beck,703-555-1212,
Recruiter, domain users, HR
• Endpoint context
10.1.1.10,crm.companyA.com,IT,Prod,SOX,PCI,Redhat 6.1, Oracle CM,
jt@companyA.com
John Smith, a member of the HR recruiting team successfully logged in as a privileged
user to an Oracle database housing the company’s CRM data, regulated by SOX & PCI

17. 17© Cloudera, Inc. All rights reserved.
Demo

18. 18© Cloudera, Inc. All rights reserved.
Open Source Collaboration
1. Collaborate with analytic, big data, and cybersecurity industry leaders
2. Share analytics with peer organizations leveraging the open data model
3. Future-proof your platform as open source community innovates at greater speed

19. 19© Cloudera, Inc. All rights reserved.
Thanks
spot.apache.org
@mark_grover