NSA for Enterprises Log Analysis Use Cases

1,171 views

Published on

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,171
On SlideShare
0
From Embeds
0
Number of Embeds
11
Actions
Shares
0
Downloads
43
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

NSA for Enterprises Log Analysis Use Cases

  1. 1. Big  Brother  for  Enterprises:     Log  Analysis  Use  Cases   Samisa  Abeysinghe   VP  Developer  Evangelism   19  Feb  2014
  2. 2. About  the  Presenter   ๏  ๏  2   Samisa  Abeysinghe   VP  Developer  Evangelism   samisa@wso2.com     Samisa  Abeysinghe,  Vice   President  of  Developer   Evangelism  joined  the  company  in   September  2005.    Prior  to  the   current  role,  Samisa  used  to  be   VP  of  Engineering  and  managed   the  development  of  WSO2   Carbon  based  product  plaOorm.    
  3. 3. About  WSO2   ๏  ๏  ๏  Global  enterprise,  founded  in  2005   by  acknowledged  leaders  in  XML,   web  services    technologies,   standards    and  open  source   Provides  only  open  source   plaOorm-­‐as-­‐a-­‐service  for  private,   public  and  hybrid  cloud   deployments   ๏  ๏  3   All  WSO2  products  are  100%  open   source  and  released  under  the   Apache  License  Version  2.0.   Is  an  AcXve  Member  of  OASIS,   Cloud  Security  Alliance,  OSGi   Alliance,  AMQP  Working  Group,   OpenID  FoundaXon  and  W3C.   Driven  by  InnovaXon   ๏  Launched  first  open  source  API   Management  soluXon  in  2012   ๏  Launched  App  Factory  in  2Q  2013   ๏  Launched  Enterprise  Store  and   first  open  source  Mobile  soluXon   in  4Q  2013  
  4. 4. What  WSO2  Deliver   4  
  5. 5. NSA like Monitoring for Your Enterprise ๏ Analyze volumes of data ๏ Address correlation complexities in analytics ๏ Off line vs Real time operations implications ๏ Some operations got to be in real time, else the value is lost ๏ Summarized data over time (and other) dimensions for analytics A “Big Brother” that keeps an eye on the whole enterprise 5  
  6. 6. Why should I bother? ๏ Deal with high volume (terra bites) of information ๏ In order to make decisions ๏ Real time & Offline ๏ Take action 6  
  7. 7. WSO2 :ONLY COMPLETE & INTEGRATED Platform ๏ Complete and integrated for ๏ Data capture ๏ Analysis: both real-time and batch ๏ Visualization ๏ Action taking business process execution Making data driven intelligence for your enterprise easy 7  
  8. 8. WSO2 Big Data Analytics Platform for Your Enterprise WSO2  Big  Data  Analy0cs   Pla4orm  for  Your  Enterprise   8  
  9. 9. Key  Elements   Data   CollecXon   Taking   AcXon   Data   Analysis   Data   VisualizaXon   9  
  10. 10. Use  Case  Scenario  1:   Monitor  your  Java  ApplicaXon  System  Logs     with  BAM  &  CEP 10  
  11. 11. & WSO2 CEP 11  
  12. 12. Overview  of  SoluXon     o  Send Log Events to o  o  o  Real time Log Event Processing o  o  With BAM analytics Visualization of Log Data o  12   With CEP Batch Processing of Log Data o  o  Business Activity Monitor (BAM) & Complex Event Processor (CEP) With Gadgets on Dashboards
  13. 13. Log  Event Publishing  (BAM)     13  
  14. 14. Event  Streams  &  AlerXng  (CEP)   WSO2 CEP 14  
  15. 15. LogEvent  Stream   q  q  Meta Data q  clientType {String} Meta Data q  tenantID {String} q  ServerName {String} q  appName {String} q  logTime {Long} q  priority {Long} q  message {String} q  logger {String} q  ip {String} q  instance {String} q  stacktrace {String} 15  
  16. 16. CEP  Query   from LogEvents [priority == "ERROR"] select message, stacktrace, serverName insert into ExceptionStream Email Body   Error Occurred in {{serverName}} – {{message}} {{stacktrace}} 16  
  17. 17. AnalyXcs  &  Batch  Processing     (BAM)   17  
  18. 18. Hive Query   CREATE EXTERNAL TABLE IF NOT EXISTS LogEventInfo (key STRING, tenantID INT,serverName STRING, appName STRING, priority STRING,logTime DOUBLE,logger STRING,message STRING) STORED BY 'org.apache.hadoop.hive.cassandra.CassandraStorageHandler' WITH SERDEPROPERTIES ( "cassandra.host" = "localhost", "cassandra.port" = "9160","cassandra.ks.name" = "EVENT_KS", "cassandra.ks.username" = "admin","cassandra.ks.password" = "admin", "cassandra.cf.name" = "log_0_AS_2014_01_23", "cassandra.columns.mapping" = ":key,payload_tenantID,payload_serverName,payload_appName, payload_priority,payload_logTime,payload_logger,payload_message" ); CREATE EXTERNAL TABLE IF NOT EXISTS Logs(tenantID INT,serverName STRING, appName STRING, priority STRING,logTime DOUBLE,logger STRING,message STRING) STORED BY 'org.wso2.carbon.hadoop.hive.jdbc.storage.JDBCStorageHandler' TBLPROPERTIES ( 'mapred.jdbc.driver.class' = 'com.mysql.jdbc.Driver', 'mapred.jdbc.url' = 'jdbc:mysql://localhost:3306/MYBAMDB', 'mapred.jdbc.username' = 'root','mapred.jdbc.password' = 'root', 'hive.jdbc.update.on.duplicate' = 'true', 'hive.jdbc.table.create.query' = 'CREATE TABLE LogEvent(tenantID INT,serverName VARCHAR(200), appName VARCHAR(200), priority VARCHAR(200),logTime DOUBLE,logger VARCHAR(800),message VARCHAR(3800))'); insert overwrite table Logs select tenantID, serverName, appName, priority, logTime, logger, message from LogEventInfo; select tenantID, serverName, appName, priority, logTime, logger, message from LogEventInfo; 18  
  19. 19. VisualizaXon -­‐  Gadgets   19  
  20. 20. Demo  …..   http://wso2.com/library/demonstrations/2014/02/screencast-monitoring-system-logs-with-wso2-business-activity-monitor/ 20  
  21. 21. Use  Case  Scenario  2:   HTTP  Log  Monitoring  With  WSO2  BAM   21  
  22. 22. HTTPD  Logs  and  Use  Cases     •  Monitor every web request information –  HTTP method –  URI –  Status code •  Monitor request properties –  Message size –  Host / IP address –  Geo location –  Date and time 22  
  23. 23. HTTPD Logs:  Scenarios   •  Publish HTTP logs to BAM from a data agent •  Split log entry for each fields and store in another big data column family •  Resolve IP addresses to geo locations •  Aggregate request for geo locations and store into a relational database (RDB) •  Visualize from gadgets 23  
  24. 24. SoluXon  Architecture     24  
  25. 25. Key  Performance  Indicator  (KPI)    Use  Cases   •  Analysing request count variation for each host •  Analysing request count variation for each hour •  Analysing request message size variation 25  
  26. 26. Final  Dashboard   26  
  27. 27. Demo  …..   http://wso2.com/library/demonstrations/2014/02/screencast-http-log-monitoring-with-wso2-business-activity-monitor/ 27  
  28. 28. Business  Model   28  
  29. 29. Call  to  acXon  page   ๏  ๏  Big  Data  Webinar  Series:   hhp://wso2.com/landing/wso2-­‐bigbrother-­‐webinar-­‐series/     ๏  WSO2  Business  AcXvity  Monitor:   hhp://wso2.com/products/business-­‐acXvity-­‐monitor/     ๏  WSO2  BAM  Docs:   hhp://docs.wso2.org/display/BAM240/WSO2+Business+AcXvity+Monitor +DocumentaXon     ๏  WSO2  Complex  Event  Processor:   hhp://wso2.com/products/complex-­‐event-­‐processor/     ๏  29   Be  your  own  NSA:  hhp://wso2.com/landing/nsa/     WSO2  CEP  Docs:   hhp://docs.wso2.org/display/CEP300/WSO2+Complex+Event+Processor +DocumentaXon    
  30. 30. Contact  us  !  

×