Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
EFFECTIVE DATA VISUALIZATION
AT WEB SCALE
PARSING CHAOS
Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confident...
Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confident...
Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confident...
Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confident...
Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confident...
Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confident...
Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confident...
Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confident...
Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confident...
Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confident...
Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confident...
Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confident...
Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confident...
Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confident...
Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confident...
Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confident...
Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confident...
Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confident...
Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confident...
Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confident...
Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confident...
Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confident...
Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confident...
Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confident...
Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confident...
Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confident...
Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confident...
Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confident...
Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confident...
Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confident...
Upcoming SlideShare
Loading in …5
×

Parsing chaos: Data Visualization at Web Scale

286 views

Published on

Data Visualization Tactics and Techniques leveraged by the BestBuy.com Site Reliability Engineering Team

Published in: Technology
  • Be the first to comment

Parsing chaos: Data Visualization at Web Scale

  1. 1. EFFECTIVE DATA VISUALIZATION AT WEB SCALE PARSING CHAOS
  2. 2. Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confidential to Best Buy. January 13, 2017 | 2 ONLINE OPERATIONS PARSING CHAOS EFFECTIVE DATA VISUALIZATION AT WEB SCALE Introduction & Overview Data Visualization: Why? Introduction to Monitoring Theory Data Visualization In Practice
  3. 3. Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confidential to Best Buy. January 13, 2017 | 3 ONLINE OPERATIONS INTRODUCTION & OVERVIEW WHO AM I? Patrick Bergstrom • Senior Manager, Site Reliability Engineering • BestBuy.com Web Operations • Chief Dashboard Jockey @IAmPatrickB
  4. 4. Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confidential to Best Buy. January 13, 2017 | 4 ONLINE OPERATIONS BESTBUY.COM WEB OPERATIONS SITE RELIABILITY ENGINEERING Site Reliability Engineering is really simple…. ....Build a website that works, and stop changing it. ......And then take a 6 month vacation -Ben Treynor, creator of SRE at Google
  5. 5. Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confidential to Best Buy. January 13, 2017 | 5 ONLINE OPERATIONS BESTBUY.COM WEB OPERATIONS WHAT DO WE DO? • 8 Web Properties • 50+ Product and Engineering Teams • 15,000 JIRA Stories • 1200 Production Releases • 150+ Deployable Web Applications • 25,000+ Requests per second • 1500+ Production Servers • 4 Datacenters Lots of opportunity for risk… • 8 Site Reliability Engineers
  6. 6. Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confidential to Best Buy. January 13, 2017 | 6 ONLINE OPERATIONS
  7. 7. Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confidential to Best Buy. January 13, 2017 | 7 ONLINE OPERATIONS WHY? DATA VISUALIZATION
  8. 8. Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confidential to Best Buy. January 13, 2017 | 8 ONLINE OPERATIONS DATA VISUALIZATION WHY • Maintain our E-Commerce platforms as High-Availability —99.99% • 52.56 Minutes of downtime per Year • Incident Response Goals: —Acknowledgement within 2 minutes —Service Restored within 20 minutes
  9. 9. Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confidential to Best Buy. January 13, 2017 | 9 ONLINE OPERATIONS DATA VISUALIZATION WHY • Primary function: Service Restoration —Support confirmation of impacted systems —Quickly identify false alarms • Secondary: Supplemental for Incident Detection —Keep an eye on ‘Hot Spots’ —Alerting Strategy detects major failures —SREs do not just stare at TVs waiting
  10. 10. Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confidential to Best Buy. January 13, 2017 | 10 ONLINE OPERATIONS DATA VISUALIZATION WHY • Tertiary: Looks cool as %^&I(** **Individual results may vary
  11. 11. Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confidential to Best Buy. January 13, 2017 | 11 ONLINE OPERATIONS DATA VISUALIZATION WHAT • Server Health — Status Codes — Response Times — CPU — Average Threads, Active Sessions — Heap — GC Count, Timing — Active Farms / Traffic Switches • Users / RUM — Concurrent Active Users — DNS Resolution/Server Connect — TTFB — DOM content/interactive — DOM complete • Non-Application Data — Test Run timing — Test Frequency — Test Failures — Severity 1 Alerts, (Pager Duty) — Notifications from third parties — Scheduled Events — Chat Message Velocity • Database: — CPU (SoftIRQ, io-wait, User, System) — I/O Wait — Network (Send, Receive, Gossip) — Disk Read/Write Latency
  12. 12. Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confidential to Best Buy. January 13, 2017 | 12 ONLINE OPERATIONS DATA VISUALIZATION HOW?
  13. 13. Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confidential to Best Buy. January 13, 2017 | 13 ONLINE OPERATIONS MONITORING THEORY DATA VISUALIZATION
  14. 14. Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confidential to Best Buy. January 13, 2017 | 14 ONLINE OPERATIONS DATA VISUALIZATION MONITORING THEORY • Status Codes! — Useful for indication of app response & performance — Good for identifying problems • Step 1: Identify the Problem — Should be handled by an Alert — SREs aren’t paid to watch TV
  15. 15. Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confidential to Best Buy. January 13, 2017 | 15 ONLINE OPERATIONS DATA VISUALIZATION MONITORING THEORY • Status Codes! — Useful for indication of app response & performance — Good for identifying problems • Step 1: Identify the Problem Symptoms — Should be handled by an Alert — SREs aren’t paid to watch TV
  16. 16. Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confidential to Best Buy. January 13, 2017 | 16 ONLINE OPERATIONS DATA VISUALIZATION MONITORING THEORY • Step 2: Actually Identify the Problem — What regions are impacted?
  17. 17. Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confidential to Best Buy. January 13, 2017 | 17 ONLINE OPERATIONS DATA VISUALIZATION MONITORING THEORY • Step 2: Actually Identify the Problem — What regions are impacted? — Did anything change? West East Blue Green
  18. 18. Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confidential to Best Buy. January 13, 2017 | 18 ONLINE OPERATIONS DATA VISUALIZATION MONITORING THEORY • Step 3: Restore Service — Undo the change West East Blue Green
  19. 19. Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confidential to Best Buy. January 13, 2017 | 19 ONLINE OPERATIONS DATA VISUALIZATION MONITORING THEORY • Response Time / Request Duration — Average vs 99th? — Average shows the exact middle of your dataset — 99th shows your top 1% (outliers) • SREs focus on Anomalies
  20. 20. Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confidential to Best Buy. January 13, 2017 | 20 ONLINE OPERATIONS DATA VISUALIZATION MONITORING THEORY • Response Time / Request Duration — Average vs 99th? — Average shows the exact middle of your dataset — 99th shows your top 1% (outliers) • SREs focus on Anomalies
  21. 21. Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confidential to Best Buy. January 13, 2017 | 21 ONLINE OPERATIONS DATA VISUALIZATION MONITORING THEORY • Response Time / Request Duration — Average vs 99th? — Average shows the exact middle of your dataset — 99th shows your top 1% (outliers) • SREs focus on Anomalies
  22. 22. Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confidential to Best Buy. January 13, 2017 | 22 ONLINE OPERATIONS DATA VISUALIZATION MONITORING THEORY
  23. 23. Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confidential to Best Buy. January 13, 2017 | 23 ONLINE OPERATIONS DATA VISUALIZATION MONITORING THEORY
  24. 24. Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confidential to Best Buy. January 13, 2017 | 24 ONLINE OPERATIONS DATA VISUALIZATION MONITORING THEORY
  25. 25. Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confidential to Best Buy. January 13, 2017 | 25 ONLINE OPERATIONS IN PRACTICE DATA VISUALIZATION
  26. 26. Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confidential to Best Buy. January 13, 2017 | 26 ONLINE OPERATIONS DATA VISUALIZATION IN PRACTICE • Rule #1: Establish Primary Dashboards — Real-estate is limited. Choosy SRE’s choose overviews. — Customer experience indicators where applicable, and KPI/Hotspot indicators for underlying app supporting it. — Don’t deep-dive, until you need to. RUM Synth RUM CPU
  27. 27. Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confidential to Best Buy. January 13, 2017 | 27 ONLINE OPERATIONS DATA VISUALIZATION IN PRACTICE • Rule #2: Not all applications are the same — Match the KPIs & hotspots for the app — Cloud and Datacenter will have different data requirements — Establish patterns for ease of recognition
  28. 28. Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confidential to Best Buy. January 13, 2017 | 28 ONLINE OPERATIONS DATA VISUALIZATION IN PRACTICE • Rule #3: Know the hot-spots of your app — Pre-build secondary dashboards to support triage — Know where you’re going to have to look — Performance testing isn’t just for the app’s benefit
  29. 29. Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confidential to Best Buy. January 13, 2017 | 29 ONLINE OPERATIONS DATA VISUALIZATION IN PRACTICE • Rule #4: Make it easy to access, share, & update your dashboards — Bookmarks, saved dashboards, etc — DISPLAY (Coming 2017!)
  30. 30. Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confidential to Best Buy. January 13, 2017 | 30 ONLINE OPERATIONS QUESTIONS? DATA VISUALIZATION
  31. 31. Confidential – Best Buy internal use only. Do not copy, publish, or distribute. The material in this document is confidential to Best Buy. January 13, 2017 | 31 ONLINE OPERATIONS QUESTIONS? DATA VISUALIZATION

×