Predictive Analytics & Visualization - Towards Data Driven Insights for Openstack Clouds

2,245 views

Published on

Slightly edited deck presented by myself and my awesome team at the Openstack Summit in May 2014 at Atlanta.

1 Comment
4 Likes
Statistics
Notes
No Downloads
Views
Total views
2,245
On SlideShare
0
From Embeds
0
Number of Embeds
22
Actions
Shares
0
Downloads
58
Comments
1
Likes
4
Embeds 0
No embeds

No notes for slide

Predictive Analytics & Visualization - Towards Data Driven Insights for Openstack Clouds

  1. 1. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Cisco Confidential© 2013 Cisco and/or its affiliates. All rights reserved. Predictive Analytics & Visualization Towards Data Driven Insights for Openstack Clouds
  2. 2. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialCisco ConfidentialCisco Confidential© 2013 Cisco and/or its affiliates. All rights reserved. Cloud/Big-data Winnovation Team Alex Holden Damian Van VuurenAiling Zhang Marc Solanas Matt Weeks Pengfei Zhang Kai Zhang Xinyuan Huang Debo Dutta Yathiraj Udupi
  3. 3. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Visualization in OpenStack today
  4. 4. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Tomorrow? Sources: 2001 A Space Oddesy, MGM. Minority Report, Twentieth Century Fox Film Corporation. Mad Men, AMC OpenStack, deploy me an app
  5. 5. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Our Approach: Predictive Analytics & Awesome Visualisation Why Predictive? ●  OpenStack has lots of moving parts ●  Rule/Model/Policy based system hard without insight Why Predictive? ●  Ops folks want to know right now! Why Visualisation? ●  Easy to observe insights by humans ●  East way to consume lots of information from loads oflogs/metrics ●  Improve user experience
  6. 6. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Cisco Confidential© 2013 Cisco and/or its affiliates. All rights reserved. Digital Exhaust = Metrics/Logs What information can we get from our cloud? Where can we get it? ●  Ceilometer ●  OpenStack Syslog ●  Project APIs (Nova, Neutron, Cinder etc) What can we do with it? ●  Extract Insights ●  Figure out what failed and why ●  Move workloads around ●  Detect attacks on the cloud ●  Physical Servers ○  CPU ○  Hardware Faults ○  OpenStack Logs ○  OpenStack Messaging Queue ●  Tenant/Project Data ○  Quotas ●  Virtual Machines ○  Config ○  CPU ○  Disk ○  Network Flows ○  Performance ○  Latency ●  Application Level ○  Application Logs ○  Application Requirements ○  Application Performance
  7. 7. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialCisco ConfidentialCisco Confidential© 2013 Cisco and/or its affiliates. All rights reserved. AVOS Dashboard
  8. 8. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential AVOS: Summary ●  Analytics and Visualization on OpenStack ●  Stateless: easy deployment ●  Client side architecture: ○  Polls some OpenStack APIs ○  Listens to the messaging bus ○  Minimum overhead ●  Provides users and service providers full insight into cloud ○  Real time info on all instances/networks etc by project and as admin ○  Historical data on performance ○  Stack-search: Search for all components in the cloud, quickly. ○  Multiple select (See combined data)
  9. 9. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential https://www.youtube.com/watch?v=vyb-Cd4VEDg AVOS Demo
  10. 10. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Bottlenecks Use Cases: Why use AVOS? Attacks
  11. 11. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Use Case: Bottlenecks
  12. 12. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Use Case: Attack Detection ●  Extract Metrics ●  Analyse Patterns ●  We can differentiate between: ○  Hadoop Traffic ○  Network Faults ○  DDoS Attack
  13. 13. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Future Plans •  Advanced Search Primitives •  Reducing the volume of logs •  Real-time error analytics •  Add to/edit the cluster •  Ability to preview changes Curvature: https://github.com/CiscoSystems/curvature
  14. 14. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Notes: OpenStack changes •  Ceilometer Better API queries, optimizing return structure Custom meters: VM to VM traffic flows •  Other suggestions/Ideas Standardization of the Python clients (several inconsistencies between method calls) Integration into horizon
  15. 15. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Conclusion Openstack Predictive Analytics/ Visibility = Reduce Ops pains We must 1) gather data 2) extract insights 3) present to ops folks with awesome UX
  16. 16. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Thank you. Please take our survey: http://goo.gl/CMAZje Coming Talk: Storage Visibility and Optimization for OpenStack: Wednesday, May 14 • 3:30pm - 4:10pm

×