Agile 2014- Metrics driven development and devops

1,223 views

Published on

There are many facets of devops, and we will spend our time in this presentation focusing on collecting and using metrics (business, application, system, etc.) and building a metrics driven culture in organizations.

We will define how we have seen devops progress in our organizations and how we’ve realized that different teams in our organizations can find common ground when teams (who have different roles) can work well together when they use metrics as the common language.

Karthik will talk about how we are using the principles from the Lean Startup to define our development cycles, sprints and using metrics to quantify how successful the products we are trying to come out with in R&D. Initially we started practicing devops on the dev and ops side of the house but realized this was still a black box to the business side of the house, so we pivoted to what our business actually understood, and that was metrics; today, we focus more on metrics (business and system level), and can fail or succeed fast to achieve our business goals faster than before.

Ernest will go into detail on how a large, mature SaaS organization uses metrics in conjunction with distributed agile development and DevOps to guide their development at scale. How much a product is used, how much each feature is used, and how much value each user gets out of it are key drivers for a business strategy - and it’s all information that’s emitted by a system. He'll show how large companies have invested time in collecting and using these metrics to guide their decisions and influence their culture.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,223
On SlideShare
0
From Embeds
0
Number of Embeds
362
Actions
Shares
0
Downloads
19
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Agile 2014- Metrics driven development and devops

  1. 1. • Ernest Mueller (@ernestmueller) • Karthik Gaekwad (@iteration1)
  2. 2. @ernestmueller @iteration1#Agile2014
  3. 3. • Senior Engineer @Signal Sciences • Previous: • 10 years building products- agile/cloud/devops teams @ernestmueller @iteration1#Agile2014
  4. 4. • Product Manager at Copperegg • Previous: • 20 years in IT – dev, ops, management @ernestmueller @iteration1#Agile2014
  5. 5. Our Goal For You Today • Empower you with new ideas to bring your organization together! • Metrics. What are they? • How to use metrics for good, as illustrated by three Epic Rap Battles of History! – Dev vs Ops (What is this… DevOps?) – Small vs Large Org – Scrum vs Kanban @ernestmueller @iteration1#Agile2014
  6. 6. @ernestmueller @iteration1#Agile2014
  7. 7. What Are Metrics? • A quantifiable measure of any component or process whose change is of interest to your business. – Business! – Application! – System! – People! – Process! – Not: Meaningless numbers! @ernestmueller @iteration1#Agile2014
  8. 8. @ernestmueller @iteration1#Agile2014
  9. 9. @ernestmueller @iteration1#Agile2014
  10. 10. 11
  11. 11. @ernestmueller @iteration1#Agile2014
  12. 12. @ernestmueller @iteration1#Agile2014
  13. 13. @ernestmueller @iteration1#Agile2014
  14. 14. @ernestmueller @iteration1#Agile2014
  15. 15. @ernestmueller @iteration1#Agile2014
  16. 16. @ernestmueller @iteration1#Agile2014
  17. 17. • Tasked with building new cloud business for the organization. • Understand how cloud technologies can impact bottom line. • Build products customers will want from the new business unit. – Read, startup inside a bigger organization Story Time: Our context @ernestmueller @iteration1#Agile2014
  18. 18. • Used ‘Lean Startup’ ideas to power new area. – Able to define an MVP (Minimum Viable Product). – Easier to define workflow for something brand new. – No confusion with existing processes. • Once we started to see value, retrofitted to other parts of the org. Lean Startup Applied @ernestmueller @iteration1#Agile2014
  19. 19. Showing progress • Initially- we had weekly progress/status meetings with stakeholders. • Cross functional team with business/marketing/engineering. @ernestmueller @iteration1#Agile2014
  20. 20. @ernestmueller @iteration1#Agile2014
  21. 21. @ernestmueller @iteration1#Agile2014
  22. 22. Metrics • Pivot: change conversations to metrics instead. • Agreed on metrics that we wanted to track – Stakeholder input • “What do you want out of this?” • “How quickly do you want this?” • …Okay, let’s measure this! @ernestmueller @iteration1#Agile2014
  23. 23. Tracked Metrics • Tracked actionable metrics (dev and business): – # Users signing up per week – # Active sessions per day/week – # of compiles sent per week – # unique data points sent per week
  24. 24. @ernestmueller @iteration1#Agile2014 Pro Tip: Metrics • Link all your metrics from one dashboard. – Business (Ex: User logins) – Dev (Ex: Performance metrics) – Ops (Ex: DB CPU Usage) • One bookmark to rule them all.
  25. 25. @ernestmueller @iteration1#Agile2014 Pro Tip: Metrics • Don’t use yet another username/password scheme. • You’ll lose your users really fast!
  26. 26. Pro Tip: Metrics • Try to use a tool that can handle different kinds of metrics. • Shoutouts: – Statsd – Datadog @ernestmueller @iteration1#Agile2014
  27. 27. End Result • Business and engineering on the same page. • Management looking at metrics without having “meetings to look at metrics”. • Became a part of the culture. • Innovate faster because different teams were in sync. @ernestmueller @iteration1#Agile2014
  28. 28. @ernestmueller @iteration1#Agile2014
  29. 29. Operations What is it? Why Do You Care? @ernestmueller @iteration1#Agile2014
  30. 30. Other Kinds Of Operations • Wikipedia quoth: • Business operations is the harvesting of value from assets owned by a business • Operations management is […] overseeing, designing, and controlling the process of production and redesigning business operations in the production of goods or services. @ernestmueller @iteration1#Agile2014
  31. 31. Technical Operations • “Operations: The New Secret Sauce” – Tim O’Reilly (2006) • Without the ability to – Release changes – Quickly respond to change – Provide a service without interruption – Operate cost effectively Your service is borked. @ernestmueller @iteration1#Agile2014
  32. 32. What Does Operations Do? • Build Servers, OS, Virtualization/Cloud • Install/Upgrade Software • Install Applications/Release Process/Move to Prod • Configure Network, Load Balancers, Storage, etc. • Security testing, reporting, and hardening • Reliability (scaling, backups) • Performance management (apps, systems) • Scalability (capacity planning to autoscaling) @ernestmueller @iteration1#Agile2014
  33. 33. What Else Does Operations Do? • Availability – Responsible for service being up • Incident Response • Fulfill Requests • Budgeting/Contracts/Cost Tracking/Reduction • Monitor all of that • Much more • So besides “they run the services,” the critical final piece of your value chain, they have access to many of the things you want metrics from
  34. 34. Code – Operations = @ernestmueller @iteration1#Agile2014
  35. 35. Story Time: Black Friday • Every year, a huge spike in usage • Uptime and performance critical to retailers during the period • Product directly contributed to conversion • Metrics crucial to plan the period, execute through the period, report how we did @ernestmueller @iteration1#Agile2014
  36. 36. @ernestmueller @iteration1#Agile2014
  37. 37. Metrics From… From: • Servers • Applications/App Logs • App Servers/Software • Network • Data Stores • Web Servers/CDNs • Client Browsers • Alerts • Tickets Using: • Open source monitoring tools (Zabbix, nagios) • SaaS (SumoLogic, PagerDuty) • JIRA • Custom (Web front end analytics w.Hadoop) • Custom (Amazon cost analytics w.GoodData) • Custom (Metrics Dashboard)
  38. 38. Many Tools Are Awful
  39. 39. YOU DECIDE
  40. 40. DEVOPS @ernestmueller @iteration1#Agile2014
  41. 41. Traditional Dev and Ops
  42. 42. What is DevOps? @ernestmueller @iteration1#Agile2014
  43. 43. What is DevOps? @ernestmueller @iteration1#Agile2014
  44. 44. Scrumming away….
  45. 45. Ready to deploy…
  46. 46. Metrics Promote DevOps • How do you get the cat inside the circle? Herding cats is hard. Some people aren’t cat people. • Metrics can be used to promote culture, understanding, and collaboration • Metrics help keep those different disciplines in sync by providing tangible collaboration points • MTTD, MTTR, performance metrics, events • Bringing all the discipline’s metrics together cover your whole value chain “code to cash”
  47. 47. Operations – Code = IT
  48. 48. IT + DevOps = ? • Many IT teams implement Agile today • They can implement DevOps too • But to do either, they have to change how they interact with others • Focus on customer’s needs not own needs; cloud/SaaS providing “competitive pressure” • Practice Theory of Constraints – embed when possible, even if you need to add some • Add devs and automate
  49. 49. SMALL ORG
  50. 50. Metrics 101: Culture of communication • Talk in terms of metrics – Builds common ground between different roles. – Understand different perspectives. – Find the best way to get everyone talking in 1 place.
  51. 51. Metrics 201 • Push your metrics into your conversation tool • Use tools that everyone likes: – IRC v/s Hipchat/slack • Integrate your metrics into a channel – “Deployment channel” in your chat
  52. 52. Culture of communication • Find a way to get people talking. • Find face to face time with stakeholders. • Metrics are that specific item to have a conversation around. • Engineering teams love IRC, but business and PM’s might not as much. • Transitioned to Slack/Hipchat (integrations and message history) • Leads to visibility and builds trust 74
  53. 53. End Result • Metrics drive conversations between everyone. • Enhances productivity. • Helped us streamline our process.
  54. 54. LARGE ORG
  55. 55. Large Mature Org • Hundreds of developers • Many teams (many goals, processes) • Distributed teams • International teams • Outsourcers • Various Weird Partner Relationships 77
  56. 56. Large Org Problems • Silos Galore • Communication Problems • Annoying Compliance Requirements • Profitability Actually Important • Less pure greenfield work – also responsibility for many existing mature systems
  57. 57. Story Time • Story Time: SaaS product, 40 Engineers, 2/3 outsourced, mostly maintenance but extreme scale (1/3 of staff were Ops) • Lots of support initiated urgent customer requests • Dev still required for features, integration/transition with newer services, bug fix, scaling • Team morale issues
  58. 58. Metrics 101 • First, add Agile. (Previously the ‘stew method’) • Basic Metrics – number of tickets (100+ in queue at any time), size of backlog (500 or so bugs and stories), rate of new inflow and completion. • Used to fix misunderstanding from upper management and correct resourcing • Next step on metrics – how to balance the support work and new work?
  59. 59. Metrics 201 Support SLA
  60. 60. Metrics 201 • Metrics 201 Velocity
  61. 61. Metrics 201 • Balancing these two metrics was the key to satisfying customers short and long term. • But it’s not an either-or - by seeing the effects of people, process, and technology changes on those metrics we drove SLA from <50% to 100% and kept velocity growing (20.. 50… 200…) • Having the metrics to focus on gave shared purpose and eased communication with the large distributed team • Experiment, see the impact, pivot.
  62. 62. Metrics 301 • Monthly “Operational Excellence (Metrics) Meeting” • Teams presented their metrics portfolio – with some variation as appropriate • Drawn from system info, app metrics, db reports, Salesforce, surveys, etc. • Keep it lean!!! • Revenue and Cost • Product Usage • Performance • Availability • Client Satisfaction • Employee Satisfaction • Quality • Security
  63. 63. Metrics 401 - A/B Testing • All features had usage measured • Feature flags would turn features on for customer subsets to measure usage, effect on conversion, etc. before committing • Sometimes you had to kill it despite work spent • Retooling could save a high profile failure • “Yes, product guy, you have to.” • Look for things metrics say you can kill – it’s the only way to stay lean long term
  64. 64. YOU DECIDE
  65. 65. @ernestmueller @iteration1#Agile2014
  66. 66. KANBAN @ernestmueller @iteration1#Agile2014
  67. 67. Here’s why… • How many meetings? • “Short planning meeting”? • How often do these go long? • Wait how long before prioritizing a feature/bug? • Role of a dedicated scrum master is a luxury. • Derailed sprints because of changing business priorities… @ernestmueller @iteration1#Agile2014
  68. 68. Why Kanban? • Limited number of WIP tasks in play. • Easier to prioritize. There is only 1 list! • Standups are simpler. • Task estimates in days versus hours (1/2 day->7 day). • Research tasks to figure out how long something may take. 94 @ernestmueller @iteration1#Agile2014
  69. 69. Kanban benefits • Kanban + CI == Solved our issue of “when to release”. Didn’t have to wait for release windows like in scrum. • Less stressful == Only x number of tasks going on at once. Easier to measure. • Velocity is awesome! 95 @ernestmueller @iteration1#Agile2014
  70. 70. Kanban Metrics • Things we track: – Visualized board (JIRA Greenhopper) – Cycle Time (How fast something gets done) – WIP (Work/Tasks in progress) – Flow diagram – %of bugs
  71. 71. SCRUM @ernestmueller @iteration1#Agile2014
  72. 72. Scrum Rules • Have used Scrum for both pure Ops and Dev+Ops teams
  73. 73. Kanban Drools • Deadlines help maintain tempo – we had multiple releases a sprint, don’t need to tie them together • You can reliably commit to a near term ETA with Scrum instead of just “when it’s done” • Scrum has a better backlog (esp. in JIRA!) • Many people “doing Kanban” are really “doing nothing”, like some doing “Agile” are really doing “cowboy coding.” Kanban takes more discipline and training than Scrum. @ernestmueller @iteration1#Agile2014
  74. 74. Scrum and Metrics • Velocity is easier for people to understand than flow diagrams @ernestmueller @iteration1#Agile2014
  75. 75. But I Hear Kanban Is Better For Ops • In a DevOps world, most Ops work SHOULD NOT be interrupt driven – it’s project work just like the devs are doing • Dev and Ops expedite work approach each other in magnitude over time assuming appropriate investment in automation • You may be thinking of “Level 1 Support” or “The Helpdesk” – that is NOT an Ops Engineer @ernestmueller @iteration1#Agile2014
  76. 76. Scrum for Ops? • Devs have to be involved in major incidents too! • Over the length of a sprint, the interrupt level evens out – my metrics show that velocity doesn’t vary more than with dev teams • You manage WIP in your scrum too @ernestmueller @iteration1#Agile2014
  77. 77. Complications Scrum Helps • Distributed teams need more communication ceremonies • Foreign/contract workers need more communication ceremonies • Same process across teams is better – in most cases other teams were using Scrum • Simple common metrics -> better collaboration • When starting from zero, Scrum was the quickest path to team continuous improvement
  78. 78. YOU DECIDE@ernestmueller @iteration1#Agile2014
  79. 79. Using Metrics For Evil @ernestmueller @iteration1#Agile2014
  80. 80. Too Many Metrics
  81. 81. Cargo Cult Metrics
  82. 82. Demand Perfection @ernestmueller @iteration1#Agile2014
  83. 83. Weaponized Metrics
  84. 84. Recap • Metrics are good - use them, be guided by them, communicate with them. @ernestmueller @iteration1#Agile2014
  85. 85. Recap • Metrics can enhance your: –Culture –Productivity –Process
  86. 86. Recap • Use them for good, not for evil.
  87. 87. @ernestmueller @iteration1 theagileadmin.com

×