Debs2010

7,297 views

Published on

Intelie presentation for the DEBS 2010 conference in Cambridge.

Published in: Technology, Education, Business
1 Comment
8 Likes
Statistics
Notes
  • Jogando.net 22
    o MELHOR SERVIDOR DE MU ONLINE DO BRASIL! Season 6 Ep. 3
    em todos os Servers. Sendo 7 servers diferenciados proporcionando sua diversão.
    Conheça também o site de Animes Cloud: www.animescloud.com ,mais de 20.000 videos online. ENTRE JÁ NO SITE : www.jogando.net/mu/ >> CADASTRE-SE E GANHE 5 DIAS DE VIP 1ª Mega Maratona Jogando.net ~> MAIS DE 30 DIAS DE EVENTOS .
    Curta nossa página no Facebook : www.facebook.com/pages/jogandonet/3710275296185
    By: talula
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total views
7,297
On SlideShare
0
From Embeds
0
Number of Embeds
12
Actions
Shares
0
Downloads
323
Comments
1
Likes
8
Embeds 0
No embeds

No notes for slide

Debs2010

  1. 1. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom HOLMES: An event-driven solution to monitor data centers through continuous queries and machine learning Pedro Henriques dos Santos Teixeira Ricardo Gomes Clemente Ronald Andreu Kaiser Denis Almeida Vieira Jr
  2. 2. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom Topics • Motivation • Use Case • The Solution • Overview • System architecture • CEP • Machine learning • CEP & Machine learning integration • Visualization and User Interface • Conclusion
  3. 3. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom Motivation
  4. 4. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom Motivation • Non-stop growing environment, dynamic • Understand our environment • Too many dependencies • Can't afford downtime
  5. 5. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom Motivation • Monitoring can be tricky • Precede the inevitable and try to avoid chaos • 1.2K servers • 14K+ monitored items • Correlation
  6. 6. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom Use Case
  7. 7. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom Use Case • Big Brother Brazil • New world record • 151 million votes in 2 days • Peaks of 13500 votes per minute (~220 v/s) • DDoS atack detected
  8. 8. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom Overview
  9. 9. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom
  10. 10. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom The System Architecture
  11. 11. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom HOLMES
  12. 12. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom System architecture – modules and its purposes • CEP module: known problems • Machine learning module: unknown problems • Visualization module: situational awareness • Storage: events history/log
  13. 13. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom CEP
  14. 14. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom CEP • Reaction to incidents in real-time is a requirement for data center monitoring • Expression of abstract rules related to the business is desirable • Correlation of events through user-defined queries
  15. 15. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom CEP - Esper • Open source CEP Implementation • Supports an EPL • High throughput, requirement in our context • Ease of embed in our application
  16. 16. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom CEP – simple example SELECT avg(response_time) FROM HTTP.win:time(5 min) events stream E5 E1 E4 E3 E2 E1 4 t.u. 3 t.u. 2 t.u. 3 t.u. 5 t.u. Ei response time 5 min ...
  17. 17. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom IfIfthe number of sessions the number of sessions increase in 10% in aa33 increase in 10% in minute window and the minute window and the average of cpu's usage of average of cpu's usage of the web farm do not the web farm do not increase in 5% and the increase in 5% and the number of slow queries in number of slow queries in the database is higher than the database is higher than 10, then we have achieved aa 10, then we have achieved database contention database contention situation. Alarm it! situation. Alarm it!
  18. 18. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom Machine learning “any signal, which is totally predictable, carries no information” - Shannon
  19. 19. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom Machine learning characteristics • FRAHST learns to detect anomalous behaviors • Unsupervised streaming algorithm • Linear complexity to the number of data streams
  20. 20. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom FRAHST, state-of-the-art For further information, see reference [12] in our paper.
  21. 21. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom Anomaly detection
  22. 22. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom CEP & Machine Learning Integration • Users choose the data streams to be correlated • CEP module aggregates events • Notifications are raised whether a rank variance is detected
  23. 23. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom Visualization and User Interface
  24. 24. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom Visualization and User Interface • Users can create Perspectives • Real-time dashboard personalizations • Events history visualization
  25. 25. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom Dashboards
  26. 26. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom
  27. 27. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom Conclusion • Successfully implementation and acceptance in a real use case • New challenges • improving situational awareness & prediction • Make creation of queries more intuitive
  28. 28. DEBS 2010 – 4th ACM International Conference on Distributed Event-Based System Cambridge, United Kingdom This presentation: http://www.slideshare.net/intelie/debs2010 Our Nagios Plugin source code: http://github.com/intelie/neb2activemq Intelligent Monitoring with Esper: http://esper.codehaus.org/tutorials/tutorial/presentations.html Denis Vieira Jr. - davieira@gmail.com Ronald Kaiser - ronald@intelie.com.br

×