The document discusses using WSO2's Big Data Analytics Platform to monitor enterprise systems and applications. It provides two use case scenarios: 1) Monitoring Java application system logs using Business Activity Monitor (BAM) and Complex Event Processor (CEP); and 2) Monitoring HTTP logs using BAM. For both scenarios, it describes how to collect, analyze, and visualize log data in real-time and batch processing. The presentation emphasizes that the WSO2 platform provides a complete, integrated solution for data capture, analysis, visualization, and action taking across data streams.

1. Big
Brother
for
Enterprises:
Log
Analysis
Use
Cases
Samisa
Abeysinghe
VP
Developer
Evangelism
19
Feb
2014

2. About
the
Presenter
๏
๏
2
Samisa
Abeysinghe
VP
Developer
Evangelism
samisa@wso2.com
Samisa
Abeysinghe,
Vice
President
of
Developer
Evangelism
joined
the
company
in
September
2005.
Prior
to
the
current
role,
Samisa
used
to
be
VP
of
Engineering
and
managed
the
development
of
WSO2
Carbon
based
product
plaOorm.

3. About
WSO2
๏
๏
๏
Global
enterprise,
founded
in
2005
by
acknowledged
leaders
in
XML,
web
services
technologies,
standards
and
open
source
Provides
only
open
source
plaOorm-­‐as-­‐a-­‐service
for
private,
public
and
hybrid
cloud
deployments
๏
๏
3
All
WSO2
products
are
100%
open
source
and
released
under
the
Apache
License
Version
2.0.
Is
an
AcXve
Member
of
OASIS,
Cloud
Security
Alliance,
OSGi
Alliance,
AMQP
Working
Group,
OpenID
FoundaXon
and
W3C.
Driven
by
InnovaXon
๏
Launched
first
open
source
API
Management
soluXon
in
2012
๏
Launched
App
Factory
in
2Q
2013
๏
Launched
Enterprise
Store
and
first
open
source
Mobile
soluXon
in
4Q
2013

4. What
WSO2
Deliver
4

5. NSA like Monitoring for
Your Enterprise
๏ Analyze volumes of data
๏ Address correlation complexities in analytics
๏ Off line vs Real time operations implications
๏ Some operations got to be in real time, else the value is lost
๏ Summarized data over time (and other) dimensions for analytics
A “Big Brother” that keeps an eye on
the whole enterprise
5

6. Why should I bother?
๏ Deal with high volume (terra bites) of information
๏ In order to make decisions
๏ Real time & Offline
๏ Take action
6

7. WSO2 :ONLY COMPLETE
& INTEGRATED Platform
๏ Complete and integrated for
๏ Data capture
๏ Analysis: both real-time and batch
๏ Visualization
๏ Action taking business process execution
Making data driven intelligence for
your enterprise easy
7

8. WSO2 Big Data Analytics
Platform for Your Enterprise
WSO2
Big
Data
Analy0cs
Pla4orm
for
Your
Enterprise
8

9. Key
Elements
Data
CollecXon
Taking
AcXon
Data
Analysis
Data
VisualizaXon
9

10. Use
Case
Scenario
1:
Monitor
your
Java
ApplicaXon
System
Logs
with
BAM
&
CEP
10

11. & WSO2 CEP
11

12. Overview
of
SoluXon
o
Send Log Events to
o
o
o
Real time Log Event Processing
o
o
With BAM analytics
Visualization of Log Data
o
12
With CEP
Batch Processing of Log Data
o
o
Business Activity Monitor (BAM) &
Complex Event Processor (CEP)
With Gadgets on Dashboards

13. Log
Event Publishing
(BAM)
13

14. Event
Streams
&
AlerXng
(CEP)
WSO2 CEP
14

15. LogEvent
Stream
q
q
Meta Data
q
clientType {String}
Meta Data
q
tenantID {String}
q
ServerName {String}
q
appName {String}
q
logTime {Long}
q
priority {Long}
q
message {String}
q
logger {String}
q
ip {String}
q
instance {String}
q
stacktrace {String}
15

16. CEP
Query
from LogEvents [priority == "ERROR"]
select message, stacktrace, serverName
insert into ExceptionStream
Email Body
Error Occurred in {{serverName}} – {{message}}
{{stacktrace}}
16

17. AnalyXcs
&
Batch
Processing
(BAM)
17

18. Hive Query
CREATE EXTERNAL TABLE IF NOT EXISTS LogEventInfo (key STRING,
tenantID INT,serverName STRING,
appName STRING, priority STRING,logTime DOUBLE,logger STRING,message STRING) STORED BY
'org.apache.hadoop.hive.cassandra.CassandraStorageHandler' WITH SERDEPROPERTIES ( "cassandra.host" =
"localhost",
"cassandra.port" = "9160","cassandra.ks.name" = "EVENT_KS",
"cassandra.ks.username"
= "admin","cassandra.ks.password" = "admin",
"cassandra.cf.name" = "log_0_AS_2014_01_23",
"cassandra.columns.mapping" =
":key,payload_tenantID,payload_serverName,payload_appName,
payload_priority,payload_logTime,payload_logger,payload_message"
);
CREATE EXTERNAL TABLE IF NOT EXISTS Logs(tenantID INT,serverName STRING,
appName STRING, priority STRING,logTime DOUBLE,logger STRING,message STRING)
STORED BY 'org.wso2.carbon.hadoop.hive.jdbc.storage.JDBCStorageHandler' TBLPROPERTIES (
'mapred.jdbc.driver.class' = 'com.mysql.jdbc.Driver',
'mapred.jdbc.url' =
'jdbc:mysql://localhost:3306/MYBAMDB',
'mapred.jdbc.username' =
'root','mapred.jdbc.password' = 'root',
'hive.jdbc.update.on.duplicate' = 'true',
'hive.jdbc.table.create.query' =
'CREATE TABLE LogEvent(tenantID INT,serverName
VARCHAR(200),
appName VARCHAR(200), priority VARCHAR(200),logTime DOUBLE,logger VARCHAR(800),message VARCHAR(3800))');
insert overwrite table Logs select tenantID, serverName, appName, priority, logTime,
logger, message from LogEventInfo;
select tenantID, serverName, appName, priority, logTime,
logger, message from LogEventInfo;
18

19. VisualizaXon -­‐
Gadgets
19

20. Demo
…..
http://wso2.com/library/demonstrations/2014/02/screencast-monitoring-system-logs-with-wso2-business-activity-monitor/
20

21. Use
Case
Scenario
2:
HTTP
Log
Monitoring
With
WSO2
BAM
21

22. HTTPD
Logs
and
Use
Cases
• Monitor every web request information
– HTTP method
– URI
– Status code
• Monitor request properties
– Message size
– Host / IP address
– Geo location
– Date and time
22

23. HTTPD Logs:
Scenarios
• Publish HTTP logs to BAM from a data agent
• Split log entry for each fields and store in another big
data column family
• Resolve IP addresses to geo locations
• Aggregate request for geo locations and store into a
relational database (RDB)
• Visualize from gadgets
23

24. SoluXon
Architecture
24

25. Key
Performance
Indicator
(KPI)
Use
Cases
• Analysing request count variation for each host
• Analysing request count variation for each hour
• Analysing request message size variation
25

26. Final
Dashboard
26

27. Demo
…..
http://wso2.com/library/demonstrations/2014/02/screencast-http-log-monitoring-with-wso2-business-activity-monitor/
27

28. Business
Model
28

29. Call
to
acXon
page
๏
๏
Big
Data
Webinar
Series:
hhp://wso2.com/landing/wso2-­‐bigbrother-­‐webinar-­‐series/
๏
WSO2
Business
AcXvity
Monitor:
hhp://wso2.com/products/business-­‐acXvity-­‐monitor/
๏
WSO2
BAM
Docs:
hhp://docs.wso2.org/display/BAM240/WSO2+Business+AcXvity+Monitor
+DocumentaXon
๏
WSO2
Complex
Event
Processor:
hhp://wso2.com/products/complex-­‐event-­‐processor/
๏
29
Be
your
own
NSA:
hhp://wso2.com/landing/nsa/
WSO2
CEP
Docs:
hhp://docs.wso2.org/display/CEP300/WSO2+Complex+Event+Processor
+DocumentaXon

30. Contact
us
!