Debs2010

DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom HOLMES: An event-driven solution to monitor data centers through continuous queries and machine learning Pedro Henriques dos Santos Teixeira Ricardo Gomes Clemente Ronald Andreu Kaiser Denis Almeida Vieira Jr

DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom Topics • Motivation • Use Case • The Solution • Overview • System architecture • CEP • Machine learning • CEP & Machine learning integration • Visualization and User Interface • Conclusion

DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom Motivation

DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom Motivation • Non-stop growing environment, dynamic • Understand our environment • Too many dependencies • Can't afford downtime

DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom Motivation • Monitoring can be tricky • Precede the inevitable and try to avoid chaos • 1.2K servers • 14K+ monitored items • Correlation

DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom Use Case

DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom Use Case • Big Brother Brazil • New world record • 151 million votes in 2 days • Peaks of 13500 votes per minute (~220 v/s) • DDoS atack detected

DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom Overview

DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom

DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom The System Architecture

DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom HOLMES

DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom System architecture – modules and its purposes • CEP module: known problems • Machine learning module: unknown problems • Visualization module: situational awareness • Storage: events history/log

DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom CEP

DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom CEP • Reaction to incidents in real-time is a requirement for data center monitoring • Expression of abstract rules related to the business is desirable • Correlation of events through user-defined queries

DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom CEP - Esper • Open source CEP Implementation • Supports an EPL • High throughput, requirement in our context • Ease of embed in our application

DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom CEP – simple example SELECT avg(response_time) FROM HTTP.win:time(5 min) events stream E5 E1 E4 E3 E2 E1 4 t.u. 3 t.u. 2 t.u. 3 t.u. 5 t.u. Ei response time 5 min ...

DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom IfIfthe number of sessions the number of sessions increase in 10% in aa33 increase in 10% in minute window and the minute window and the average of cpu's usage of average of cpu's usage of the web farm do not the web farm do not increase in 5% and the increase in 5% and the number of slow queries in number of slow queries in the database is higher than the database is higher than 10, then we have achieved aa 10, then we have achieved database contention database contention situation. Alarm it! situation. Alarm it!

DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom Machine learning “any signal, which is totally predictable, carries no information” - Shannon

DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom Machine learning characteristics • FRAHST learns to detect anomalous behaviors • Unsupervised streaming algorithm • Linear complexity to the number of data streams

DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom FRAHST, state-of-the-art For further information, see reference [12] in our paper.

DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom Anomaly detection

DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom CEP & Machine Learning Integration • Users choose the data streams to be correlated • CEP module aggregates events • Notifications are raised whether a rank variance is detected

DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom Visualization and User Interface

DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom Visualization and User Interface • Users can create Perspectives • Real-time dashboard personalizations • Events history visualization

DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom Dashboards

DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom

DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom Conclusion • Successfully implementation and acceptance in a real use case • New challenges • improving situational awareness & prediction • Make creation of queries more intuitive

DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom This presentation: http://www.slideshare.net/intelie/debs2010 Our Nagios Plugin source code: http://github.com/intelie/neb2activemq Intelligent Monitoring with Esper: http://esper.codehaus.org/tutorials/tutorial/presentations.html Denis Vieira Jr. - davieira@gmail.com Ronald Kaiser - ronald@intelie.com.br

Debs2010

by Intelie on Jul 14, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

1 Embed 1

Statistics

Debs2010 — Presentation Transcript