Optimizing IT Costs & Services With Big Data (Little Effort!) - Case Studies - #Pink13

568 views

Published on

IT organizations have a wealth of Service Management and Service Delivery tools, processes and metrics that typically exist in relative isolation. This session will present detailed real-life examples of how existing tools and metrics can be brought together using big data techniques to optimize costs and performance of IT environments.

  • Be the first to comment

  • Be the first to like this

Optimizing IT Costs & Services With Big Data (Little Effort!) - Case Studies - #Pink13

  1. 1. Optimizing IT Costs & Services with Big Data, Little Effort… David Wagner TeamQuest AdvocateTeamQuest and the TeamQuest logo are registered trademarks in the US, EU and elsewhere.All other trademarks and service marks are the property of their respective owners.Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
  2. 2. Agenda • Why? • Big Data: conceptual overview • 2013 Capacity Management 101: – History – Goals – Obstacles • New “Big Data” approaches – Concepts – Case Study ValueCopyright © 2012 TeamQuest Corporation. All Rights Reserved.
  3. 3. Why does TeamQuest Exist? • We passionately believe always having and using the right amount of resources is a societal imperative – Anything less is failure – Anything more is wasteful • 20+ years sole focus – ensuring our customers can continuously and automatically perform at their utmost level of efficiency – ensuring business service performance, conserving scarce resources, saving money and improving productivity • We call this: IT Service OptimizationCopyright © 2012 TeamQuest Corporation. All Rights Reserved.
  4. 4. What this Presentation is… and is not! • IS: – Applying Big data approaches to Capacity Management • Faster and larger value • More scalable – New ways to think about optimization beyond ITIL Capacity Management • Include ITIL Service Management and Delivery • Not just technology anymore • Is NOT! – A Primer on Big Data or a Big Data “how to” Presentation • Hadoop ecosystem deep dive, etc…Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
  5. 5. Big Data at 50,000 feet… • Big Data is about: data  actionable information – Plethora of existing sources • Technology • Business (Sales, Marketing, …) • Service (Transactions, SLAs, …) – Learning new insights from “old” data – Key is Analytics • Deep • Wide • Adaptable • But… Optimizing costs with Capacity Management?Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
  6. 6. Technology Approaches • Data Access and Aggregation – Build huge “data marts” (aka: Data Warehousing) – Integrate with multiple different data sources • Technology (e.g. Server, Network, Storage, etc.) • Service (Catalog, Metrics, Tickets, etc.) • Business (KPIs, Plans, Transactions, etc.) • Implement Analytics against/across – Flexible and adaptive – Turn data within, into actionable information across • But… Capacity Management???Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
  7. 7. 2013 Capacity Management 101 - History • Answering “what if” questions… – Change in technology, demand, etc… impact? – Focus on Optimizing Server Cost versus Performance • Extremely Technology-centric – Servers, Mainframes – Occasionally Storage or Network – in isolation • Big Value and Return, but also effort – Highly trained staff – Required building a central, massive datamart (CMIS) – Scalability of Staff, Tools, …, PoliticsCopyright © 2012 TeamQuest Corporation. All Rights Reserved.
  8. 8. 2013 Capacity Management – Goals: What • Maintain traditional value, and add – Optimize – Amplify – Accelerate • Increase Business relevance – Valuable predictive analytics in business and service context – Optimize Efficiency • Virtualization and Cloud Scale to everything – Many to many inter-relationships; Capacity criticalCopyright © 2012 TeamQuest Corporation. All Rights Reserved.
  9. 9. 2013 Capacity Management 101 – Goals: How • Integrate and Analyze across multiple sources – Technology (e.g. Server, Network, Storage, etc.) – Service (Catalog, Metrics, Tickets, etc.) – Business (KPIs, Plans, Transactions, etc.) • Single pane of “Analytic Glass” – Ability to tie together, correlate, and operate across • Tear down the wall! – Don’t force reinvention… or data duplication! – Flexible and adaptive – Turn data within, into actionable information acrossCopyright © 2012 TeamQuest Corporation. All Rights Reserved.
  10. 10. 2013 Capacity Management 101 - Obstacles • Data Access and Aggregation – Building huge “data marts” (fka: Data Warehousing) • Complexity = (data ETL) x (# sources) x (maintenance effort) • Compliance: Data duplication, privacy, audit, etc… • Costly and time consuming • Implementing Analytics against/across – General purpose BI Analytics for Capacity? – Traditional Performance/Capacity for General Purpose? • “Big Data” + ITIL = Optimized Capacity Management?Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
  11. 11. Capacity Management with ITIL 2011 • Service Strategy – Financial management • Service Design – Service Level and Availability management • Service Transition – Asset, Change and Configuration Management • Service Operations – Service Desk – Application and IT operations – Event, Incident, Problem • Or, in simpler terms… – Integrate Capacity across ITIL V2: Service Support and Service Delivery!Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
  12. 12. Optimized Capacity Management • Leverage the data (and tools) you have! – Don’t reinvent or reimplement • Quickly and easily with True Federation – Use existing data/tools already in place – Don’t force data duplication, ETL – Capacity Analysis across data sources • Key ITIL discipline metrics amplify Capacity Management Value – Strategy  factor financials – Design  factor Service Levels, technology performance – Transition  track business and technology changes – Operations  factor Service risks, multiple technologiesCopyright © 2012 TeamQuest Corporation. All Rights Reserved.
  13. 13. Integrated Case Study Walkthrough • ITIL: Strategy – Capacity Management integrated with Financial costing/reporting • ITIL: Design – Capacity Management integrated with Risk Registry • ITIL: Transition – Includes integration with Asset and Configuration Management • ITIL: Operations – Integration with Service Desk – Operations  factor Service risks, multiple technologiesCopyright © 2012 TeamQuest Corporation. All Rights Reserved.
  14. 14. Very Large Bank As an IT Shop: • Operate tens of thousands of servers • Every server platform under the sun • Manage dozens of data centers • Huge mainframe with many thousands of MIPS • Thousands of VMs • Thousands of VDIs & Citrix • Many Petabytes of storageCopyright © 2012 TeamQuest Corporation. All Rights Reserved. 14
  15. 15. Seamless data integration & analysis 1. All capacity/performance data 2. All platforms, OS’s, … 3. Configuration data 4. Change records 5. Risk registryCopyright © 2012 TeamQuest Corporation. All Rights Reserved. 15
  16. 16. Deliverable: Fully Automated Application Report We need: 1. Risk detection and tracking 2. Risk reporting 3. Actionable information Reporting has to be: • Automated • Repeatable • Human-readable – financials, business terms, not “speeds and feeds”Copyright © 2012 TeamQuest Corporation. All Rights Reserved. 16
  17. 17. Analysis Overview Application and Configuration from Service Catalog and CMDBCopyright © 2012 TeamQuest Corporation. All Rights Reserved. 17
  18. 18. Usage Patterns Time Series data from Performance and Event ManagementCopyright © 2012 TeamQuest Corporation. All Rights Reserved. 18
  19. 19. Service Desk and Risk managementCopyright © 2012 TeamQuest Corporation. All Rights Reserved. 19
  20. 20. Existing Capacity Issues Scaleably ID Possible Copyright © 2012 TeamQuest Corporation. All Rights Reserved. 20
  21. 21. ID Possible Future Capacity Risks Copyright © 2012 TeamQuest Corporation. All Rights Reserved. 21
  22. 22. Fixed Costs / Variable Costs - Method Variable Costs Source: wikipedia.org Fixed CostsCopyright © 2012 TeamQuest Corporation. All Rights Reserved. 22
  23. 23. Capacity Management + Strategy (Financials) Fixed/Variable Cost server0009b01a - Excess Capacity Report Produced by the Server Capacity & Performance Management (SCPM) Team Analysis Period: August 01 2010 to August 31 2010 Run Time: 4:09 PM September 27 2010 (8 seconds) Purpose: To analyze the systems current resource consumption and compute the available headroom based on a fixed/variable costs methodology and our rules-of-thumb. This report also attempts to determine the nearest bottlenecks, from a consumption perspective. server0009b01a: Maximum Growth Capability by Resource server0009b01a: Top 10 PIDs Name Growth Vaule NAME PIDGROWTH SLOPE MINCPU AVGCPU MAXCPU CPU RunQ Length Growth 2.15 System:4 17.54 0.00 0.06 0.09 5.18 Disk - 0 4.41 NTRtScan:1660 29.88 -0.00 0.00 0.02 3.01 Memory Utilization Growth 5.48 beasvc:1080 47.72 -0.00 0.00 0.14 1.89 FS - C: 10.61 svchost:840 184.41 -0.00 0.03 0.07 0.52 Virtual Memory Growth 20.27 svchost:872 213.22 0.00 0.05 0.08 0.47 CPU Growth 38.10 TmListen:2160 763.86 -0.00 0.00 0.01 0.12 Net In 100MB - NIC1 260.82 python:1788 848.86 0.00 0.00 0.03 0.11 Net Out 100MB - NIC1 349.39 wmiprvse:268 987.95 -0.00 0.00 0.04 0.09 Net In 1GB - NIC1 2608.18 wmiprvse:2228 1322.98 0.00 0.02 0.04 0.09 Net Out 1GB - NIC1 3493.92 wmiprvse:2044 1328.88 -0.00 0.02 0.05 0.09 23Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
  24. 24. Capacity Management + Strategy (Financials)Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
  25. 25. Capacity Management + Strategy (Financials)Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
  26. 26. VM Optimization Analysis • Thousands of VMs • Some too small • Some too big • Some idle • Which ones? • What size should they be?Copyright © 2012 TeamQuest Corporation. All Rights Reserved. 26
  27. 27. Physical to Virtual AnalysisCopyright © 2012 TeamQuest Corporation. All Rights Reserved. 27
  28. 28. Capacity Optimization Candidates Total Virtual Machines Idle Virtual Machines Oversized Virtual Machines 22 3 13Idle Virtual Machines Recommende Avg % Max Max % Average Avg % Max % CPU Avg % Total Recommende d SSO vCPUs Max % CPU CPU Memory Memory Memory Memory Ready CPU Memory d SSO vCPUs Memory in Ready Used Util Used Util GBCLUSTER0019V019 4 0 0 0 0 4096 0 0 0 0 1 2CLUSTER0019V024 2 0 0 0 0 2000 0 0 0 0 1 2CLUSTER0019V029-OLD_DO_NOT_USE 4 0 0 0 0 4096 0 0 0 0 1 2Oversized Virtual Machines Avg % Max Max % Average Avg % Recommend Recommended Max % CPU Avg % Total vCPUs Max % CPU CPU Memory Memory Memory Memory ed SSO SSO Memory in Ready CPU Memory Ready Used Util Used Util vCPUs GBCLUSTER0019V001 2 40 5 9 1 2044 1921 94 1729 85 1 4CLUSTER0019V003 2 30 10 7 2 2048 1895 93 1737 85 1 4CLUSTER0019V004 2 45 51 3 3 2048 1914 93 1333 65 2 4CLUSTER0019V005 2 41 17 8 2 2048 1955 95 1716 84 2 4CLUSTER0019V006 2 45 41 2 2 2048 1963 96 1510 74 2 4CLUSTER0019V008 2 27 23 2 2 2048 1860 91 1232 60 1 4CLUSTER0019V013 2 30 40 3 3 2048 1845 90 1326 65 1 4CLUSTER0019V014 2 30 36 3 2 2048 1834 90 1286 63 1 4CLUSTER0019V018 2 32 30 3 2 2000 1843 92 1581 79 1 4CLUSTER0019V029-REAL 4 42 30 5 4 4096 3612 88 2951 72 4 8CLUSTER0019V030 2 47 23 2 2 2048 1881 92 1387 68 2 4CLUSTER0019v009 2 43 14 2 1 4096 3117 76 2258 55 2 4CLUSTER0019v010 2 30 18 2 1 4096 3102 76 2151 53 1 4 Copyright © 2012 TeamQuest Corporation. All Rights Reserved. 28
  29. 29. Delivered: • Repeatable processes • Quicker analysis • Powerful • FlexibleCopyright © 2012 TeamQuest Corporation. All Rights Reserved. 29

×