More Related Content Similar to Framework and Product Comparison for Big Data Log Analytics and ITOA (20) More from Kai Wähner (20) Framework and Product Comparison for Big Data Log Analytics and ITOA 1. Big Data Log Analytics and
IT Operations Analytics (ITOA)
with Splunk, TIBCO LogLogic and the Open Source “ELK Stack”
Kai Wähner
kwaehner@tibco.com
@KaiWaehner
www.kai-waehner.de
LinkedIn / Xing Please connect!
2. 3
Rapid Growth in Machine Big Data Challenges IT
© Copyright 2000-2015 TIBCO Software Inc.
?
3. 4
When a Threat or Opportunity is Discovered in Your Logs…
© Copyright 2000-2015 TIBCO Software Inc.
• Can you issue a single search across all your
machine data- regardless of source or type?
• Can you set an alert that would trigger from any
source in your enterprise?
• What about „predictive monitoring“?
• Are you storing all of your logs for enough time to answer the
question “What happened?” a week from now? How about a year
from now?
4. Key Messages
– Log Analytics enables IT Operations Analytics for Machine Data
– Correlation of Events is the Key for Added Business Value
– Log Management is complementary to other Big Data Components
5. Agenda
– Real World Use Cases
– Introduction to Log Analytics
– Market Overview
– Live Demo
– Relation to other Big Data Components
6. Agenda
– Real World Use Cases
– Introduction to Log Analytics
– Market Overview
– Live Demo
– Relation to other Big Data Components
7. 8
© Copyright 2000-2015 TIBCO Software Inc.
Real World Use Cases
Infrastructure
• Log Management
– Applications
– SOA
– Microservices
– SaaS
• Transaction Tracing
• Root Cause Analysis
• Visual Analytics on Machine Data
Competitive Undermining
• Filtering / Cost Avoidance Solution
IT Operations
• Troubleshooting Connectivity
• Outage Troubleshooting
• Application Monitoring / Tracking
• Service Level Confirmation for IT Outsourcing
Security
• Centralized Log/Event Management Platform
• Security
• Fraud Detection
Compliance
• PCI Compliance
• Retention Compliance
• Service Level Confirmation for IT Outsourcing
8. Agenda
– Real World Use Cases
– Introduction to Log Analytics
– Market Overview
– Live Demo
– Relation to other Big Data Components
9. Service Level
Assurance Compliance Security
Business
ActivityIT Operations
Problem: Point to Point Architecture
Cloud
Same
information
being stored
multiples
times = more
HW, more
cost
Redundant
solutions
create
network
burden by
collecting
same data
multiple
times
© Copyright 2000-2015 TIBCO Software Inc.
10. Solution: Operation Intelligence Platform
Cloud
Cloud
© Copyright 2000-2015 TIBCO Software Inc.
Service Level
Assurance Compliance Security
Business
ActivityIT Operations
Log Management
11. Key Benefits of the Operational Intelligence Platform
© Copyright 2000-2015 TIBCO Software Inc.
SLA Compliance Security Identity IT Ops
LogLogic
Cloud
Cloud
12. How an Operation Intelligence Platform Works
© Copyright 2000-2015 TIBCO Software Inc.
Collect Data from
Any Source
Device Logs
Web Logs
Application & DB Logs
Configuration Files
OS Metrics
Sensor Data
INGEST
13. How an Operation Intelligence Platform Works
© Copyright 2000-2015 TIBCO Software Inc.
Collect Data from
Any Source
Device Logs
Web Logs
Application & DB Logs
Configuration Files
OS Metrics
Sensor Data
Make Unstructured
Data Usable
Normalize
Enrich
Transform
Index
Aggregate
INGEST OPERATIONALIZE
14. How an Operation Intelligence Platform Works
© Copyright 2000-2015 TIBCO Software Inc.
INGEST OPERATIONALIZE ANALYZE
Collect Data from
Any Source
Device Logs
Web Logs
Application & DB Logs
Configuration Files
OS Metrics
Sensor Data
Make Unstructured
Data Usable
Normalize
Enrich
Transform
Index
Aggregate
Gain Actionable
Insight
Search
Report
Alert
Correlate
Visualize
15. 34
Characteristics of Log Management Solutions
© Copyright 2000-2015 TIBCO Software Inc.
Data Sources
– Log information (standard protocols like TCP, UDP, File, Syslog)
– All events (logs, messaging, streams, ...)
– Extendable plugins (connectors, SDK, API)
Features
– Collect, parse, correlate, search, report, forward, etc.
– Store and index
– Query Lanaguage (SQL, Custom) sliding windows, correlations, etc.
– Retention
– Compliance Templates
Frequency
– Historical data
– Near Real Time Processing (seconds or minutes)
Deployment Options
– On-premise vs. Cloud (SaaS)
– Open Source vs. Commercial
– Software vs. Hardware Appliance
Pricing
– Free (open source) vs. CPU-based vs. Volume-based
Be careful here: IoT... Data grows exponentialy
16. Agenda
– Real World Use Cases
– Introduction to Log Analytics
– Market Overview
– Live Demo
– Relation to other Big Data Components
17. 36
Security information and event management (SIEM)
© Copyright 2000-2015 TIBCO Software Inc.
SIEM is a specific part of Log Analytics
focusing on Security:
• Threat management: Early detection of targeted
attacks and data breaches
• Compliance: Collect, store, analyze and report
on log data for incident response, forensics and
regulatory compliance
• Aggregates event data produced by security
devices, network infrastructures, systems and
applications
Log Analytics handles all kinds of use
cases, not focusing on security.
http://www.gartner.com/document/3097022
https://www-01.ibm.com/marketing/iwm/dre/signup?source=swg-WW_Security_Organic&S_PKG=ov37658&cm_mmc=Blog_SI-_-Sec_Int-_-Organic-_-IBM-is-a-leader-again-in-2015-gartner-magic-quadrant-for-SIEM
SIEM is out-of-scope for this presentation!
18. 37
Market Analysis
* Market size data from various sources (sources in notes)
Rapidly Emerging and Evolving, Encompasses Many Segments
Traditional: Log Management, IT Operations Monitoring (ITOM), Security (SIEM)
Current: IT Operations Analytics (ITOA), Application Performance Management (APM)
Future: DevOps & Continuous Improvement
Segment CAGR Incumbents Challengers
Log Management
15%
Splunk, TIBCO LogLogic, etc. Open Source (Graylog, “ELK Stack”)
SIEM RSA, ArcSight, LogRhythm Splunk, MSSPs (Managed Security Service Provider)
ITOA (1.6B) 100% TIBCO Unity, Splunk, SumoLogic, AppDynamics, NewRelic
APM (2.9B) 10% AppDynamics, NewRelic
ITOM (19B) 4% IBM, CA, BMC, MS, HP AppDynamics, NewRelic, Chef, Puppet, Docker, CloudFoundry
(2.9B)
19. 38
Alternatives for Log Analytics
Time
to
Market
Log Analytics
Product
Middleware Suite
(includes Log Analytics Product)
Slow Fast
Log Analytics
Framework
IncludesIncludes
© Copyright 2000-2015 TIBCO Software Inc.
20. 39
Alternatives for Log Management
© Copyright 2000-2015 TIBCO Software Inc.
Open Source Closed Source
SaaS
On Premise
(no complete list)
21. 40
Alternatives for Log Management
© Copyright 2000-2015 TIBCO Software Inc.
Open Source Closed Source
SaaS
On Premise
(no complete list)
Open Source Framework
22. 41
Alternatives for Log Analytics
Time
to
Market
Log Analytics
Product
Middleware Suite
(includes Log
Analytics Product)
Slow Fast
Log Analytics
Framework
© Copyright 2000-2015 TIBCO Software Inc.
Library (Java, .NET, Python)
Operators (Collect, Filter, Sort, Aggregate, Alert)
Scalability (Horizontal and Vertical, Fail Over)
Connectivity (Standards, Technologies, Products)
User Interface (Basic Monitoring and Reporting)
23. 42
ELK Stack (Logstash, Elasticsearch, Kibana)
© Copyright 2000-2015 TIBCO Software Inc.
Characteristics
• Data Sources
• Features
• Frequency
• Deployment Options
• Pricing
Facts
• Combination of Open Source Frameworks
– Complex setup and usage (coding and configuration)
• Targeted for developers
– Mainly focused on helping developers detect and fix errors in their apps
– Entirely open source, i.e. free to use
– Commerical support available
– Combination of different mature frameworks
• Less enterprise-focused
– Very basic user interface
– Based on ElasticSearch, Logstash and Kibana
– Plenty of connectors + easy to extend (with coding)
– Missing extensive reporting and analytics
24. 43
graylog
© Copyright 2000-2015 TIBCO Software Inc.
Characteristics
• Data Sources
• Features
• Frequency
• Deployment Options
• Pricing
Facts
• Combination of Open Source Frameworks
– Complex setup and usage (coding and configuration)
• Targeted for developers
– Mainly focused on helping developers detect and fix errors in their apps
– Entirely open source, i.e. free to use
– Commerical support available
– Young solution (1.0 GA in 2015) – not as mature as others yet
• Less enterprise-focused
– Very basic user interface
– Based on MongoDB, ElasticSearch and Apache Kafka
– Marketplace for connectors + easy to extend (with coding)
– Missing extensive reporting and analytics
25. 44
Alternatives for Log Management
© Copyright 2000-2015 TIBCO Software Inc.
Open Source Closed Source
SaaS
On Premise
(no complete list)
SaaS Cloud Service
26. 45
Alternatives for Log Analytics
Time
to
Market
Log Analytics
Product
Middleware Suite
(includes Log
Analytics Product)
Slow Fast
Log Analytics
Framework
© Copyright 2000-2015 TIBCO Software Inc.
Library
Operators
Scalability
Connectivity
User Interface
Visual Configuration (Analysis, Correlation, Alerting)
Simulation (Feed Testing, Test Generation)
User Interface (Advanced Monitoring, Reporting, Analytics)
Maturity (product, 24h support, consulting)
27. 46
papertrail
© Copyright 2000-2015 TIBCO Software Inc.
Facts
• Easy setup and very simple to use
• Targeted for developers
– „Very small“ free version available (100MB/month)
– Cheap pricing, e.g. 1GB/month: 5 USD; 1000GB/month: 875 USD
• Less enterprise-focused
– Stripped down and basic log analyzer
– Mostly text-based
– User interface is very similar to looking at a log on your machine
– No advanced integrations, predictive or reporting capabilities
• SaaS
– Upload (masses of) data to the cloud
– Worse latency than on-premise solutions
– Efforts to anonymize sensitive data
Characteristics
• Data Sources
• Features
• Frequency
• Deployment Options
• Pricing
28. 47
loggly
© Copyright 2000-2015 TIBCO Software Inc.
Facts
• Easy setup and very simple to use
– Custom performance and DevOps dashboards
• Targeted for developers and DevOps
– Pricing from 50 USD to some thousand USD
– Feature-limited free version available (200MB/day)
• Less enterprise-focused
– Focus especially on logs from application servers
– Anything beyond that has to be built
– Find and fix operational problems
– Primary use cases are for troubleshooting / customer support scenarios
• SaaS
– Upload (masses of) data to the cloud
– Worse latency than on-premise solutions
– Efforts to anonymize sensitive data
Characteristics
• Data Sources
• Features
• Frequency
• Deployment Options
• Pricing
29. 48
sumologic
© Copyright 2000-2015 TIBCO Software Inc.
Characteristics
• Data Sources
• Features
• Frequency
• Deployment Options
• Pricing
Facts
• Easy setup and simple to use
• Targeted for developer, security teams, business
– Pricing from 90 USD to some thousand USD
– Feature-limited free version available (500MB/day)
• Most enterprise-focused SaaS product
– Founded as „Splunk for the Cloud“
– Most feature-rich SaaS solution
– Many features of „enterprise grade solutions“
• SaaS
– Upload (masses of) data to the cloud
– Worse latency than on-premise solutions
– Efforts to anonymize sensitive data
30. 49
Alternatives for Log Management
© Copyright 2000-2015 TIBCO Software Inc.
Open Source Closed Source
SaaS
On Premise
(no complete list)
Enterprise Product
31. 50
Splunk
© Copyright 2000-2015 TIBCO Software Inc.
Characteristics
• Data Sources
• Features
• Frequency
• Deployment Options
• Pricing
Facts
• Complex setup (especially for larger scale)
• Simple to use for the end user
• Targeted for all use cases (including SIEM)
– Not just for log files, but also other events / messaging
– „Enterprise Pricing“ - Very High pricing (for medium and high volume)
– No access to your data if limit is reached! (contrary to other vendors)
• Enterprise Class
– Market leader
– Most feature-rich solution
– Available as SaaS offering
– Moving into ITOA market
– No hardware appliance (just via partner „SBOX“)
– Just log analytics, no complete middleware suite
32. 51
Alternatives for Log Analytics
Time
to
Market
Log Analytics
Product
Middleware Suite
(includes Log
Analytics Product)
Slow Fast
Log Analytics
Framework
© Copyright 2000-2015 TIBCO Software Inc.
Library
Operators
Scalability
Connectivity
User Interface
Visual Configuration
Simulation
Advanced User Interface
Maturity
Out-of-the-Box Integration and Support
(Messaging, ESB, MDM, etc.)
33. 52
IBM QRadar
© Copyright 2000-2015 TIBCO Software Inc.
Characteristics
• Data Sources
• Features
• Frequency
• Deployment Options
• Pricing
Facts
• Complex setup
• Simple to use for the end user
• Targeted for all use cases (including SIEM)
– Not just for log files, but also other events / messaging
– „Enterprise Pricing“ - High pricing (for medium and high volume)
• Enterprise Class
– Part of a complete middlware suite
– Very feature-rich solution
– Available as SaaS offering
– Available as hardware appliance
– Moving into ITOA market
34. 53
TIBCO LogLogic
© Copyright 2000-2015 TIBCO Software Inc.
Characteristics
• Data Sources
• Features
• Frequency
• Deployment Options
• Pricing
Facts
• Easy setup (small and large scale)
• Simple to use for the end user
– Powerful user interface
– Not as powerful as Splunk or IBM QRadar
• Targeted for all use cases
– Not just for log files, but also other events / messaging
– „Enterprise Pricing“ - Low costs compared to competitors
– „Always on“ – even after limit is reached
• Enterprise Class
– Part of a complete middlware suite
– Most advanced analytics (via TIBCO Spotfire add-on)
– Available as hardware appliance
– Ready for ITOA (via TIBCO LogLogic Unity)
35. 54
Spoilt for Choice?
© Copyright 2000-2015 TIBCO Software Inc.
Does it make sense
to combine different Log
Analytics solutions?
36. 55
Example: TIBCO LogLogic - A Splunk Management Solution
© Copyright 2000-2015 TIBCO Software Inc.
http://www.tibco.de/assets/blt0da0bc2ea7d5b9b7/solution-brief-tibco-loglogic-splunk-management-solution.pdf
37. 56
Conclusion - Market Analysis
© Copyright 2000-2015 TIBCO Software Inc.
Log Management
• SaaS Easy to setup and use, but cloud cons (not flexible, public cloud)
• Open Source Free and extendable, but coding / config instead of tooling
• Enterprise Most feature-rich and powerful tooling, but more expensive
IT Operations Analytics (ITOA)
• Enterprise vendors entering this market these days
– Extending existing solutions
• Focus on complex correlations, real time processing, predictive monitoring
38. 57
Market Analysis
* Market size data from various sources (sources in notes)
Rapidly Emerging and Evolving, Encompasses Many Segments
Traditional: Log Management, IT Operations Monitoring (ITOM), Security (SIEM)
Current: IT Operations Analytics (ITOA), Application Performance Management (APM)
Future: DevOps & Continuous Improvement
Segment CAGR Incumbents Challengers
Log Management
15%
Splunk, TIBCO LogLogic, etc. Open Source (Graylog, “ELK Stack”)
SIEM RSA, ArcSight, LogRhythm Splunk, MSSPs (Managed Security Service Provider)
ITOA (1.6B) 100% TIBCO Unity, Splunk, SumoLogic, AppDynamics, NewRelic
APM (2.9B) 10% AppDynamics, NewRelic
ITOM (19B) 4% IBM, CA, BMC, MS, HP AppDynamics, NewRelic, Chef, Puppet, Docker, CloudFoundry
(2.9B)
39. 58
IT Operations Analytics (ITOA)
© Copyright 2000-2015 TIBCO Software Inc.
http://www.evolven.com/blog/gartner-analysts-have-high-expectations-for-it-operations-analytics.html
40. Agenda
– Real World Use Cases
– Introduction to Log Analytics
– Market Overview
– Live Demo
– Relation to other Big Data Components
42. Agenda
– Real World Use Cases
– Introduction to Log Analytics
– Market Overview
– Live Demo
– Relation to other Big Data Components
43. 64
When to use Log Analytics
Time of
Action
Historical
Data
Near
Real Time
Real Time Predictive
IT Operations Analytics (ITOA)
Log Management
Data Warehouse Streaming Analytics
Data Discovery
Hadoop (Variety of different Frameworks)
Log Analytics
44. 65
Relation to other Big Data Components
© Copyright 2000-2015 TIBCO Software Inc.
• Data Warehouse
– Historical data
– Only structured data
– Reporting
• Apache Hadoop
– Historical and near real time data
– All data
– Storage and Analytics (e.g. MapReduce, Spark)
• NoSQL
– Specific Storage (graph, document, key/value, ...)
– Search (e.g. ElasticSearch)
• Stream Processing
– Especially real time data
• Predictive Analytics
– R, Machine Learning, SAS, etc.
– Combined with the others!
Log
Analytics
Forward
Forward
Parse, Filter, Structure, Forward
Parse, Filter, Structure, Forward
Parse, Filter, Structure, Forward
45. 66
Log Management / ITOA vs. Hadoop and Log Collectors
© Copyright 2000-2015 TIBCO Software Inc.
Why not use just Hadoop? You can also store and analyze all data on its cluster!
Why not just use Log Collectors and send data directly without Log Analytics “in the middle”?
• In general: Fluentd, Logstash,
• Hadoop specific: Apache Flume or Apache Kafka
DIFFERENTIATORS OF LOG MANAGEMENT / IT OPERATIONS ANALYTICS
• Integrated solution for data analysis (tooling, consulting, support)
• Built exactly for these use cases (Log Management, ITOA)
• Involves data indexing, data processing (querying) and data visualization by means of dashboards
and other tools
• Tooling for Easy-of-Use and Time-to-Market
• Graphical user interface for operational intelligence
47. – Log Analytics enables IT Operations Analytics for Machine Data
– Correlation of Events is the Key for Added Business Value
– Log Management is complementary to other Big Data Components
Key Messages