Framework and Product Comparison for Big Data Log Analytics and ITOA

Big Data Log Analytics and
IT Operations Analytics (ITOA)
with Splunk, TIBCO LogLogic and the Open Source “ELK Stack”
Kai Wähner
kwaehner@tibco.com
@KaiWaehner
www.kai-waehner.de
LinkedIn / Xing  Please connect!

3
Rapid Growth in Machine Big Data Challenges IT
© Copyright 2000-2015 TIBCO Software Inc.
?

4
When a Threat or Opportunity is Discovered in Your Logs…
• Can you issue a single search across all your
machine data- regardless of source or type?
• Can you set an alert that would trigger from any
source in your enterprise?
• What about „predictive monitoring“?
• Are you storing all of your logs for enough time to answer the
question “What happened?” a week from now? How about a year
from now?

Key Messages
– Log Analytics enables IT Operations Analytics for Machine Data
– Correlation of Events is the Key for Added Business Value
– Log Management is complementary to other Big Data Components

Agenda
– Real World Use Cases
– Introduction to Log Analytics
– Market Overview
– Live Demo
– Relation to other Big Data Components

8
Real World Use Cases
Infrastructure
• Log Management
– Applications
– SOA
– Microservices
– SaaS
• Transaction Tracing
• Root Cause Analysis
• Visual Analytics on Machine Data
Competitive Undermining
• Filtering / Cost Avoidance Solution
IT Operations
• Troubleshooting Connectivity
• Outage Troubleshooting
• Application Monitoring / Tracking
• Service Level Confirmation for IT Outsourcing
Security
• Centralized Log/Event Management Platform
• Security
• Fraud Detection
Compliance
• PCI Compliance
• Retention Compliance
• Service Level Confirmation for IT Outsourcing

Service Level
Assurance Compliance Security
Business
ActivityIT Operations
Problem: Point to Point Architecture
Cloud
Same
information
being stored
multiples
times = more
HW, more
cost
Redundant
solutions
create
network
burden by
collecting
same data
multiple
times

Solution: Operation Intelligence Platform
Cloud
Cloud
Service Level
Assurance Compliance Security
Business
ActivityIT Operations
Log Management

Key Benefits of the Operational Intelligence Platform
SLA Compliance Security Identity IT Ops
LogLogic
Cloud
Cloud

How an Operation Intelligence Platform Works
Collect Data from
Any Source
Device Logs
Web Logs
Application & DB Logs
Configuration Files
OS Metrics
Sensor Data
INGEST

Collect Data from
Any Source
Device Logs
Web Logs
Configuration Files
OS Metrics
Sensor Data
Make Unstructured
Data Usable
Normalize
Enrich
Transform
Index
Aggregate
INGEST OPERATIONALIZE

INGEST OPERATIONALIZE ANALYZE
Collect Data from
Any Source
Device Logs
Web Logs
Configuration Files
OS Metrics
Sensor Data
Make Unstructured
Data Usable
Normalize
Enrich
Transform
Index
Aggregate
Gain Actionable
Insight
Search
Report
Alert
Correlate
Visualize

34
Characteristics of Log Management Solutions
Data Sources
– Log information (standard protocols like TCP, UDP, File, Syslog)
– All events (logs, messaging, streams, ...)
– Extendable plugins (connectors, SDK, API)
Features
– Collect, parse, correlate, search, report, forward, etc.
– Store and index
– Query Lanaguage (SQL, Custom)  sliding windows, correlations, etc.
– Retention
– Compliance Templates
Frequency
– Historical data
– Near Real Time Processing (seconds or minutes)
Deployment Options
– On-premise vs. Cloud (SaaS)
– Open Source vs. Commercial
– Software vs. Hardware Appliance
Pricing
– Free (open source) vs. CPU-based vs. Volume-based
 Be careful here: IoT... Data grows exponentialy

36
Security information and event management (SIEM)
SIEM is a specific part of Log Analytics
focusing on Security:
• Threat management: Early detection of targeted
attacks and data breaches
• Compliance: Collect, store, analyze and report
on log data for incident response, forensics and
regulatory compliance
• Aggregates event data produced by security
devices, network infrastructures, systems and
applications
Log Analytics handles all kinds of use
cases, not focusing on security.
http://www.gartner.com/document/3097022
https://www-01.ibm.com/marketing/iwm/dre/signup?source=swg-WW_Security_Organic&S_PKG=ov37658&cm_mmc=Blog_SI-_-Sec_Int-_-Organic-_-IBM-is-a-leader-again-in-2015-gartner-magic-quadrant-for-SIEM
SIEM is out-of-scope for this presentation!

37
Market Analysis
* Market size data from various sources (sources in notes)
Rapidly Emerging and Evolving, Encompasses Many Segments
Traditional: Log Management, IT Operations Monitoring (ITOM), Security (SIEM)
Current: IT Operations Analytics (ITOA), Application Performance Management (APM)
Future: DevOps & Continuous Improvement
Segment CAGR Incumbents Challengers
Log Management
15%
Splunk, TIBCO LogLogic, etc. Open Source (Graylog, “ELK Stack”)
SIEM RSA, ArcSight, LogRhythm Splunk, MSSPs (Managed Security Service Provider)
ITOA (1.6B) 100% TIBCO Unity, Splunk, SumoLogic, AppDynamics, NewRelic
APM (2.9B) 10% AppDynamics, NewRelic
ITOM (19B) 4% IBM, CA, BMC, MS, HP AppDynamics, NewRelic, Chef, Puppet, Docker, CloudFoundry
(2.9B)

38
Alternatives for Log Analytics
Time
to
Market
Log Analytics
Product
Middleware Suite
(includes Log Analytics Product)
Slow Fast
Log Analytics
Framework
IncludesIncludes

39
Alternatives for Log Management
Open Source Closed Source
SaaS
On Premise
(no complete list)

40
SaaS
On Premise
(no complete list)
Open Source Framework

41
Time
to
Market
Log Analytics
Product
Middleware Suite
(includes Log
Analytics Product)
Slow Fast
Log Analytics
Framework
Library (Java, .NET, Python)
Operators (Collect, Filter, Sort, Aggregate, Alert)
Scalability (Horizontal and Vertical, Fail Over)
Connectivity (Standards, Technologies, Products)
User Interface (Basic Monitoring and Reporting)

42
ELK Stack (Logstash, Elasticsearch, Kibana)
Characteristics
• Data Sources
• Features
• Frequency
• Deployment Options
• Pricing
Facts
• Combination of Open Source Frameworks
– Complex setup and usage (coding and configuration)
• Targeted for developers
– Mainly focused on helping developers detect and fix errors in their apps
– Entirely open source, i.e. free to use
– Commerical support available
– Combination of different mature frameworks
• Less enterprise-focused
– Very basic user interface
– Based on ElasticSearch, Logstash and Kibana
– Plenty of connectors + easy to extend (with coding)
– Missing extensive reporting and analytics

43
graylog
Characteristics
• Data Sources
• Features
• Frequency
• Pricing
Facts
• Combination of Open Source Frameworks
– Complex setup and usage (coding and configuration)
– Mainly focused on helping developers detect and fix errors in their apps
– Entirely open source, i.e. free to use
– Commerical support available
– Young solution (1.0 GA in 2015) – not as mature as others yet
– Very basic user interface
– Based on MongoDB, ElasticSearch and Apache Kafka
– Marketplace for connectors + easy to extend (with coding)
– Missing extensive reporting and analytics

44
SaaS
On Premise
(no complete list)
SaaS Cloud Service

45
Time
to
Market
Log Analytics
Product
Middleware Suite
(includes Log
Analytics Product)
Slow Fast
Log Analytics
Framework
Library
Operators
Scalability
Connectivity
User Interface
Visual Configuration (Analysis, Correlation, Alerting)
Simulation (Feed Testing, Test Generation)
User Interface (Advanced Monitoring, Reporting, Analytics)
Maturity (product, 24h support, consulting)

46
papertrail
Facts
• Easy setup and very simple to use
– „Very small“ free version available (100MB/month)
– Cheap pricing, e.g. 1GB/month: 5 USD; 1000GB/month: 875 USD
– Stripped down and basic log analyzer
– Mostly text-based
– User interface is very similar to looking at a log on your machine
– No advanced integrations, predictive or reporting capabilities
• SaaS
– Upload (masses of) data to the cloud
– Worse latency than on-premise solutions
– Efforts to anonymize sensitive data
Characteristics
• Data Sources
• Features
• Frequency
• Pricing

47
loggly
Facts
• Easy setup and very simple to use
– Custom performance and DevOps dashboards
• Targeted for developers and DevOps
– Pricing from 50 USD to some thousand USD
– Feature-limited free version available (200MB/day)
– Focus especially on logs from application servers
– Anything beyond that has to be built
– Find and fix operational problems
– Primary use cases are for troubleshooting / customer support scenarios
• SaaS
Characteristics
• Data Sources
• Features
• Frequency
• Pricing

48
sumologic
Characteristics
• Data Sources
• Features
• Frequency
• Pricing
Facts
• Easy setup and simple to use
• Targeted for developer, security teams, business
– Pricing from 90 USD to some thousand USD
– Feature-limited free version available (500MB/day)
• Most enterprise-focused SaaS product
– Founded as „Splunk for the Cloud“
– Most feature-rich SaaS solution
– Many features of „enterprise grade solutions“
• SaaS

49
SaaS
On Premise
(no complete list)
Enterprise Product

50
Splunk
Characteristics
• Data Sources
• Features
• Frequency
• Pricing
Facts
• Complex setup (especially for larger scale)
• Simple to use for the end user
• Targeted for all use cases (including SIEM)
– Not just for log files, but also other events / messaging
– „Enterprise Pricing“ - Very High pricing (for medium and high volume)
– No access to your data if limit is reached! (contrary to other vendors)
• Enterprise Class
– Market leader
– Most feature-rich solution
– Available as SaaS offering
– Moving into ITOA market
– No hardware appliance (just via partner „SBOX“)
– Just log analytics, no complete middleware suite

51
Time
to
Market
Log Analytics
Product
Middleware Suite
(includes Log
Analytics Product)
Slow Fast
Log Analytics
Framework
Library
Operators
Scalability
Connectivity
User Interface
Visual Configuration
Simulation
Advanced User Interface
Maturity
Out-of-the-Box Integration and Support
(Messaging, ESB, MDM, etc.)

52
IBM QRadar
Characteristics
• Data Sources
• Features
• Frequency
• Pricing
Facts
• Complex setup
• Targeted for all use cases (including SIEM)
– „Enterprise Pricing“ - High pricing (for medium and high volume)
– Part of a complete middlware suite
– Very feature-rich solution
– Available as SaaS offering
– Available as hardware appliance
– Moving into ITOA market

53
TIBCO LogLogic
Characteristics
• Data Sources
• Features
• Frequency
• Pricing
Facts
• Easy setup (small and large scale)
– Powerful user interface
– Not as powerful as Splunk or IBM QRadar
• Targeted for all use cases
– „Enterprise Pricing“ - Low costs compared to competitors
– „Always on“ – even after limit is reached
– Part of a complete middlware suite
– Most advanced analytics (via TIBCO Spotfire add-on)
– Available as hardware appliance
– Ready for ITOA (via TIBCO LogLogic Unity)

54
Spoilt for Choice?
Does it make sense
to combine different Log
Analytics solutions?

55
Example: TIBCO LogLogic - A Splunk Management Solution
http://www.tibco.de/assets/blt0da0bc2ea7d5b9b7/solution-brief-tibco-loglogic-splunk-management-solution.pdf

56
Conclusion - Market Analysis
Log Management
• SaaS  Easy to setup and use, but cloud cons (not flexible, public cloud)
• Open Source  Free and extendable, but coding / config instead of tooling
• Enterprise  Most feature-rich and powerful tooling, but more expensive
• Enterprise vendors entering this market these days
– Extending existing solutions
• Focus on complex correlations, real time processing, predictive monitoring

57
Market Analysis
* Market size data from various sources (sources in notes)
Rapidly Emerging and Evolving, Encompasses Many Segments
Traditional: Log Management, IT Operations Monitoring (ITOM), Security (SIEM)
Current: IT Operations Analytics (ITOA), Application Performance Management (APM)
Future: DevOps & Continuous Improvement
Segment CAGR Incumbents Challengers
Log Management
15%
Splunk, TIBCO LogLogic, etc. Open Source (Graylog, “ELK Stack”)
SIEM RSA, ArcSight, LogRhythm Splunk, MSSPs (Managed Security Service Provider)
ITOA (1.6B) 100% TIBCO Unity, Splunk, SumoLogic, AppDynamics, NewRelic
APM (2.9B) 10% AppDynamics, NewRelic
ITOM (19B) 4% IBM, CA, BMC, MS, HP AppDynamics, NewRelic, Chef, Puppet, Docker, CloudFoundry
(2.9B)

58
http://www.evolven.com/blog/gartner-analysts-have-high-expectations-for-it-operations-analytics.html

Papertrail (SaaS), ELK Stack (Open Source) and TIBCO LogLogic / Unity (Enterprise) in Action…
Live Demo

64
When to use Log Analytics
Time of
Action
Historical
Data
Near
Real Time
Real Time Predictive
Log Management
Data Warehouse Streaming Analytics
Data Discovery
Hadoop (Variety of different Frameworks)
Log Analytics

65
Relation to other Big Data Components
• Data Warehouse
– Historical data
– Only structured data
– Reporting
• Apache Hadoop
– Historical and near real time data
– All data
– Storage and Analytics (e.g. MapReduce, Spark)
• NoSQL
– Specific Storage (graph, document, key/value, ...)
– Search (e.g. ElasticSearch)
• Stream Processing
– Especially real time data
• Predictive Analytics
– R, Machine Learning, SAS, etc.
– Combined with the others!
Log
Analytics
Forward
Forward
Parse, Filter, Structure, Forward

66
Log Management / ITOA vs. Hadoop and Log Collectors
Why not use just Hadoop? You can also store and analyze all data on its cluster!
Why not just use Log Collectors and send data directly without Log Analytics “in the middle”?
• In general: Fluentd, Logstash,
• Hadoop specific: Apache Flume or Apache Kafka
DIFFERENTIATORS OF LOG MANAGEMENT / IT OPERATIONS ANALYTICS
• Integrated solution for data analysis (tooling, consulting, support)
• Built exactly for these use cases (Log Management, ITOA)
• Involves data indexing, data processing (querying) and data visualization by means of dashboards
and other tools
• Tooling for Easy-of-Use and Time-to-Market
• Graphical user interface for operational intelligence

– Log Analytics enables IT Operations Analytics for Machine Data
– Correlation of Events is the Key for Added Business Value
– Log Management is complementary to other Big Data Components
Key Messages

Questions?
Kai Wähner
kwaehner@tibco.com
@KaiWaehner
www.kai-waehner.de
LinkedIn / Xing  Please connect!

Framework and Product Comparison for Big Data Log Analytics and ITOA

More Related Content

What's hot

Similar to Framework and Product Comparison for Big Data Log Analytics and ITOA

More from Kai Wähner

Recently uploaded

Framework and Product Comparison for Big Data Log Analytics and ITOA