Uploaded on

Intelie presentation for the DEBS 2010 conference in Cambridge.

Intelie presentation for the DEBS 2010 conference in Cambridge.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
  • Jogando.net 22
    o MELHOR SERVIDOR DE MU ONLINE DO BRASIL! Season 6 Ep. 3
    em todos os Servers. Sendo 7 servers diferenciados proporcionando sua diversão.
    Conheça também o site de Animes Cloud: www.animescloud.com ,mais de 20.000 videos online. ENTRE JÁ NO SITE : www.jogando.net/mu/ >> CADASTRE-SE E GANHE 5 DIAS DE VIP 1ª Mega Maratona Jogando.net ~> MAIS DE 30 DIAS DE EVENTOS .
    Curta nossa página no Facebook : www.facebook.com/pages/jogandonet/3710275296185
    By: talula
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
6,353
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
309
Comments
1
Likes
8

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom HOLMES: An event-driven solution to monitor data centers through continuous queries and machine learning Pedro Henriques dos Santos Teixeira Ricardo Gomes Clemente Ronald Andreu Kaiser Denis Almeida Vieira Jr
  • 2. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom Topics • Motivation • Use Case • The Solution • Overview • System architecture • CEP • Machine learning • CEP & Machine learning integration • Visualization and User Interface • Conclusion
  • 3. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom Motivation
  • 4. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom Motivation • Non-stop growing environment, dynamic • Understand our environment • Too many dependencies • Can't afford downtime
  • 5. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom Motivation • Monitoring can be tricky • Precede the inevitable and try to avoid chaos • 1.2K servers • 14K+ monitored items • Correlation
  • 6. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom Use Case
  • 7. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom Use Case • Big Brother Brazil • New world record • 151 million votes in 2 days • Peaks of 13500 votes per minute (~220 v/s) • DDoS atack detected
  • 8. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom Overview
  • 9. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom
  • 10. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom The System Architecture
  • 11. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom HOLMES
  • 12. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom System architecture – modules and its purposes • CEP module: known problems • Machine learning module: unknown problems • Visualization module: situational awareness • Storage: events history/log
  • 13. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom CEP
  • 14. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom CEP • Reaction to incidents in real-time is a requirement for data center monitoring • Expression of abstract rules related to the business is desirable • Correlation of events through user-defined queries
  • 15. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom CEP - Esper • Open source CEP Implementation • Supports an EPL • High throughput, requirement in our context • Ease of embed in our application
  • 16. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom CEP – simple example SELECT avg(response_time) FROM HTTP.win:time(5 min) events stream E5 E1 E4 E3 E2 E1 4 t.u. 3 t.u. 2 t.u. 3 t.u. 5 t.u. Ei response time 5 min ...
  • 17. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom IfIfthe number of sessions the number of sessions increase in 10% in aa33 increase in 10% in minute window and the minute window and the average of cpu's usage of average of cpu's usage of the web farm do not the web farm do not increase in 5% and the increase in 5% and the number of slow queries in number of slow queries in the database is higher than the database is higher than 10, then we have achieved aa 10, then we have achieved database contention database contention situation. Alarm it! situation. Alarm it!
  • 18. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom Machine learning “any signal, which is totally predictable, carries no information” - Shannon
  • 19. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom Machine learning characteristics • FRAHST learns to detect anomalous behaviors • Unsupervised streaming algorithm • Linear complexity to the number of data streams
  • 20. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom FRAHST, state-of-the-art For further information, see reference [12] in our paper.
  • 21. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom Anomaly detection
  • 22. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom CEP & Machine Learning Integration • Users choose the data streams to be correlated • CEP module aggregates events • Notifications are raised whether a rank variance is detected
  • 23. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom Visualization and User Interface
  • 24. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom Visualization and User Interface • Users can create Perspectives • Real-time dashboard personalizations • Events history visualization
  • 25. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom Dashboards
  • 26. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom
  • 27. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom Conclusion • Successfully implementation and acceptance in a real use case • New challenges • improving situational awareness & prediction • Make creation of queries more intuitive
  • 28. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom This presentation: http://www.slideshare.net/intelie/debs2010 Our Nagios Plugin source code: http://github.com/intelie/neb2activemq Intelligent Monitoring with Esper: http://esper.codehaus.org/tutorials/tutorial/presentations.html Denis Vieira Jr. - davieira@gmail.com Ronald Kaiser - ronald@intelie.com.br