This document discusses using big data analytics to enhance security. It begins by defining big data analytics and describing security trends like the evolution from intrusion detection systems to security information and event management (SIEM) to next-generation SIEM using big data analytics. An example of an advanced persistent threat is provided. The document then discusses integrating security analytics with open source tools like SQRRL and Prelert. Finally, it covers how to apply these concepts by determining what security-related data can be collected and two options for implementing big data analytics in a security program.

1. Big Data Analytics to Enhance Security
Anapat Pipatkitibodee
Technical Manager
The First NIDA Business Analytics and Data Sciences Contest/Conference
วันที่ 1-2 กันยายน 2559 ณ อาคารนวมินทราธิราช สถาบันบัณฑิตพัฒนบริหารศาสตร์
https://businessanalyticsnida.wordpress.com
https://www.facebook.com/BusinessAnalyticsNIDA/
ใช้ Big Data มาเพิ่มความปลอดภัยได้อย่างไร
Big Data Analytics
Security Trends
Example Security Attack
Integrated Security Analytics with Open Source
How to apply?
นวมินทราธิราช 3001 วันที่ 1 กันยายน 2559 16.30-17.00 น.

2. Big Data Analytics to
Enhance Security
Anapat Pipatkitibodee
Technical Manager
STelligence Company Limited
anapat.p@stelligence

3. Agenda
• Big Data Analytics
• Security Trends
• Example Security Attacks
• Integrated Security Analytics with Open Source
• How to Apply ?

4. Big Data Analytics

5. Everyone is Claiming Big Data

6. Traditional vs Big Data

7. Drivers of Big Data
• About 80% of the world’s data are semi-structured or unstructured.

8. Open Source Tools in Big Data
• Hadoop ecosystem
• NoSQL database

9. Apache Hadoop Stack
Reference:
Hadoop Essentials
by Swizec Teller

10. https://whatsthebigdata.com/2016/02/08/big-data-landscape-2016/

11. Big Data Analytics
• The process of examining large data
sets containing a variety of data types
i.e., big data.
• Big Data analytics enables
organizations to analyze a mix of
structured, semi-structured, and
unstructured data in search of
valuable information and insights.

12. Security Trends

13. Data Analytics for Intrusion Detection
• 1st generation: Intrusion detection systems
• 2nd generation: Security information and
event management (SIEM)

14. Limitation of Traditional SIEMs
Storing and retaining a large quantity of data was not economically
feasible.
Normalization & datastore schema reduces data
Traditional tools did not leverage Big Data technologies.
Closed platform with limited customization & integration options

15. Security Trend from Y2015 to Y2016
Fireeye M-Trends Report 2016

16. Security Trend from Y2015 to Y2016
• Threats are hard to investigate
Fireeye M-Trends Report 2016

17. All Data is Security Relevant = Big Data
Servers
Storage
DesktopsEmail Web
Transaction
Records
Network
Flows
DHCP/ DNS
Hypervisor
Custom
Apps
Physical
Access
Badges
Threat
Intelligence
Mobile
CMDB
Intrusion
Detection
Firewall
Data Loss
Prevention
Anti-
Malware
Vulnerability
Scans
Traditional
Authentication

18. Data Analytics for Intrusion Detection
• 1st generation: Intrusion detection systems
• 2nd generation: Security information and
event management (SIEM)
• 3rd generation: Big Data analytics in
security (Next generation SIEM)

19. Example Security Attacks

20. Advanced Persistent Threats
• Advanced
• The attack can cope with traditional security solutions
• In many cases is based on Zero-day vulnerabilities
• Persistent
• Attack has a specific goal
• Remain on the system as long as the attack goal is not met.
• Threat
• Collect and steal information-Confidentiality.
• Make the victim's system unavailable-Availability.
• Modify the victim's system data-Integrity.

21. Example of Advanced Threat Activities
HTTP (web) session to
command & control
server
Remote control,
Steal data,
Persist in company,
Rent as botnet
WEB
Conduct
Business
Create additional
environment
Gain Access
to systemTransaction
.pdf
.pdf executes & unpacks malware
overwriting and running “allowed” programs
Svchost.exeCalc.exe
Attacker hacks website
Steals .pdf files
Web
Portal.pdf
Attacker creates
malware, embed in
.pdf,
Emails
to the target
MAIL
Read email, open
attachment
Threat
intelligence
Auth - User
Roles
Host
Activity/Security
Network
Activity/Security

22. Link Events Together
Threat intelligence
Auth - User Roles,
Corp Context
Host
Activity/Security
Network
Activity/Security
WEB
Conduct
Business
Create additional
environment
Gain Access
to systemTransaction
MAIL
.pdf Svchost.exeCalc.exe
Events that
contain link to file
Proxy log
C2
communication
to blacklist
How was
process
started?
What created the
program/process
?
Process
making C2
traffic
Web
Portal.pdf

23. Correlated Security Log
Aug 08 06:09:13 acmesep01.acmetech.com Aug 09 06:17:24 SymantecServer acmesep01: Virus found,Computer
name: ACME-002,Source: Real Time Scan,Risk name: Hackertool.rootkit,Occurrences: 1,C:/Documents and
Settings/smithe/Local Settings/Temp/evil.tmp,"""",Actual action: Quarantined,Requested action: Cleaned, time:
2009-01-23 03:19:12,Inserted: 2009-01-23 03:20:12,End: 2009-01-23 03:19:12,Domain: Default,Group: My
CompanyACME Remote,Server: acmesep01,User: smithe,Source computer: ,Source IP: 10.11.36.20
Aug 08 08:26:54 snort.acmetech.com {TCP} 10.11.36.20:5072 -> 10.11.36.26:443 itsec snort[18774]:
[1:100000:3] [Classification: Potential Corporate Privacy Violation] Credit Card Number Detected in Clear
Text [Priority: 2]:
20130806041221.000000Caption=ACME-2975EBAdministrator Description=Built-in account for
administering the computer/domainDomain=ACME-2975EB InstallDate=NULLLocalAccount = IP: 10.11.36.20
TrueName=Administrator SID =S-1-5-21-1715567821-926492609-725345543 500SIDType=1
Status=Degradedwmi_ type=UserAccounts
Sources
All three occurring within a 24-hour period
Source IP
Data Loss
Default Admin Account
Malware Found
Time Range
Intrusion
Detection
Endpoint
Security
Windows
Authentication
Source IP
Source IP

24. Incident Analysis & Investigation
Search historically - back in time Watch for new evidence
Related
evidence
from other
security
devices

25. Integrated Security Analytics with
Open Source

26. SQRRL Solution
https://sqrrl.com/

27. Anomaly detection in Visualizing
https://sqrrl.com/

28. Prelert Behavioral Analytics
for the Elastic Stack
http://info.prelert.com/

29. Prelert Behavioral Analytics
for the Elastic Stack
http://info.prelert.com/

30. How to Apply ?

31. Determining Data That Can Be Collected
Threat
intelligence
Auth - User
Roles
Service
Host
Network
Network Security Through Data Analysisby Michael S
CollinsPublished by O'Reilly Media, Inc., 2014
• Third-party Threat Intel
• Open source blacklist
• Internal threat intelligence
• Firewall
• IDS / IPS
• Web Proxy
• Vulnerability scanners
• VPNs
• Netflow
• TCP Collector
• OS logs
• Patching
• File Integrity
• Endpoint (AV/IPS/FW)
• Malware detection
• Logins, Logouts log
• Active Directory
• LDAP
• AAA, SSO
• Application logs
• Audit log
• Service / Process

32. Option 1 : Replace All Solution
• Data sent to new Big Data
Analytic Platform
• Big Data Analytic Platform
• Static Visualizations /
Reports
• Threat detection, alerts,
workflow, compliance
• Incident
investigations/forensics
• Non-security use cases
Big Data Analytic Platform
Raw data
Alerts
Static
Visualizations
Forensics / Search
Interface

33. Option 2 : Big Data to Traditional SIEM
• Data sent to both system
• Big Data Analytic Platform
• Incident
investigations/forensics
• Non-security use cases
• Traditional SIEM
• Static Visualizations /
Reports
• Threat detection, alerts,
workflow, compliance
Big Data Analytic
Platform
Raw data
Forensics / Search
Interface
SIEM
Alerts
Static
Visualizations
Connectors

34. Factors for evaluating
Big Data Security Analytics Platforms
Factors for Evaluating Open Source
• Scalable data ingestion HDFS
• Unified data management platform Cassandra / Accumulo
• Support for multiple data types Ready to Customized
• Real time Spark / Strom
• Security analytic tools No
• Compliance reporting No
• Easy to deploy and manage Manage many 3rd Party
• Flexible search, report and create
new correlation rule
No