Metrics-Driven DevOps: Delivering Software Like the Unicorn
•Download as PPTX, PDF•
1 like•285 views
This document contains slides from a presentation about metrics-driven DevOps. Some of the key points discussed include measuring performance at each stage of development and deployment, using metrics to detect regressions and problems, monitoring services and users in production, and building and delivering applications with a metrics-driven pipeline. Contact information is provided for the presenter Andreas Grabner and for learning more about Dynatrace tools and techniques.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (18)
Similar to Metrics-Driven DevOps: Delivering Software Like the Unicorn
Similar to Metrics-Driven DevOps: Delivering Software Like the Unicorn (20)
More from Beyond20
More from Beyond20 (16)
Recently uploaded
Recently uploaded (20)
Metrics-Driven DevOps: Delivering Software Like the Unicorn
- 1. #b20Con ITSM DEVOPS CONFERENCE Metrics-Driven DevOps Andreas Grabner (@grabnerandi) - andreas.grabner@dynatrace.com “Delivering Software like the Unicorns”
- 3. #b20Con
- 9. #b20Con
- 10. #b20Con
- 11. #b20Con
- 12. #b20Con
- 13. #b20Con
- 14. #b20Con
- 15. #b20Con
- 16. #b20Con
- 18. #b20Con SMART ROOVES Alert BEFORE It is too late!
- 19. #b20Con
- 20. #b20Con
- 21. #b20Con
- 22. #b20Con
- 23. #b20Con The stuff we did when we were a Start Up and we All were Devs, Testers and Ops
- 24. #b20Con
- 25. #b20Con “IN YOUR FACE” DATA! https://dynatrace.github.io/ufo/
- 27. #b20Con
- 28. #b20Con
- 29. #b20Con Analyzing SQLs by Type (SELECT, INSERT, UPDATE,DELETE) Which Types take how long? When do you have spikes? When do we see # of SQL spikes? Spikes related to INSERT, UPDATE, DELETE? Any job running?
- 30. #b20Con
- 31. #b20Con
- 32. #b20Con
- 33. #b20Con 700 deployments / YEAR 10 + deployments / DAY 50 – 60 deployments / DAY Every 11.6 SECONDS
- 34. #b20Con NOT ONLY FAST DELIVERED, BUT ALSO DELIVERING FAST! -1000ms +2% Response Time Conversions -1000ms +10% +100ms -1%
- 35. #b20Con 60% Rate performance/response time as the #1 mobile app expectation - ahead of features and functionality -
- 36. #b20Con
- 37. #b20Con It‘s not about blindly giving everyone Ops power to deploy changes only tested locally
- 38. #b20Con It‘s not about blind automation of pushing more bad code on new stacks through a pipeline
- 39. #b20Con
- 40. #b20Con
- 41. #b20Con
- 42. #b20Con 282! Objects on that page9.68MB Page Size 8.8s Page Load Time Most objects are images delivered from your main domain Very long Connect time (1.8s) to your CDN DevOps Deployment Example 1: Online Casino
- 43. #b20Con
- 44. #b20Con 2015201420xx Response Time 2016+ 1) Started as a small project 2) Slowly growing user base 3) Expanding to new markets – 1st performance degradation! 4) Adding more markets – performance becomes a business impact Users 4) Potentially start loosing users Online Sports Club Search Service
- 45. #b20Con EARLY 2015: MONOLITHIC APP Can‘t scale vertically endlessly! 2.68s Load Time 94.09% CPU Bound
- 46. #b20Con PROPOSAL: SERVICE APPROACH! Front End to Cloud Scale Backend in Containers!
- 47. #b20Con TESTING THE BACKEND SERVICE ALONE SCALES WELL 7:00 a.m. Low Load and Service running on minimum redundancy 12:00 p.m. Scaled up service during peak load with failover of problematic node 7:00 p.m. Scaled down again to lower load and move to different geo location
- 51. #b20Con SINGLE SEARCH QUERY END-TO-END 26.7s Load Time 5kB Payload 33! Service Calls 99kB - 3kB for each call! 171! Total SQL Count Architecture Violation Direct access to DB from frontend service
- 52. #b20Con THE FIXED END-TO-END USE CASE “Re-Architect” vs. “Migrate” to Service-Orientation 2.5s (vs 26.7) 5kB Payload 1! (vs 33!) Service Call 5kB(vs 99) Payload! 3! (vs 177) Total SQL Count
- 53. #b20Con
- 54. #b20Con YOU MEASURE IT! FROM DEV TO OPS
- 55. #b20Con METRICS FROM & FOR DEV (TO) OPS Build 17 testNewsAlert OK testSearch OK Build # Use Case Stat # API Calls # SQL Payload CPU 1 5 2kb 70ms 1 3 5kb 120ms Use Case Tests and Monitors Service & App Metrics Build 26 testNewsAlert OK testSearch OK Build 25 testNewsAlert OK testSearch OK 1 4 1kb 60ms 34 171 104kb 550ms Ops #ServInst Usage RT 1 0.5% 7.2s 1 63% 5.2s 1 4 1kb 60ms 2 3 10kb 150ms 1 0.6% 4.2s 5 75% 2.5s Build 35 testNewsAlert - testSearch OK - - - - 2 3 10kb 150ms - - - 8 80% 2.0s Re-architecture into „Services“ + Performance Fixes Scenario: Monolithic App with 2 Key Features
- 56. #b20Con #1: Analyzing every Unit, Integration & REST API test #2: Key Architectural Metrics for each test #3: Detecting regression based on measure per Check-in #1: STOP BAD BUILDS IN CI
- 57. #b20Con #2: MONITOR YOUR SERVICES/USERS IN PROD #1: Usage Tip: UEM Conversion! #2: Load vs Response Tip: See unusual spikes #3: Architectural Metrics DB, Exceptions, Web Service Calls
- 58. #b20Con #3: METRICS PER SERVICE IN OPS # SQLs per Search # RESTs per Search Spot bad Deployment? Payload per Search
- 59. #b20Con #4: UX ANALYSIS BASED ON UEM DATA #1: Are they using the features we built? #2: Is there a difference between Premium and Normal users? #3: Does Performance have a Behavior Impact?
- 60. #b20Con BUILD & DELIVER APPS THAT CAN EAT THE WORLD! WITH A METRICS-DRIVEN PIPELINE!
- 62. #b20Con #3 You Automate/Virtualize #2 You Measure Dev(to)Ops #1 You Build It, You Run It! #4 You API vs You App #5 Apply for Unicorn Status
- 63. #b20Con Questions •Slides: slideshare.net/grabnerandi •Get Tools: bit.ly/dtpersonal •YouTube Tutorials: bit.ly/dttutorials •Contact Me: agrabner@dynatrace.com •Follow Me: @grabnerandi •Read More: blog.dynatrace.com
- 64. #b20Con QUESTIONS? Slides: slideshare.net/grabnerandi Get Tools: bit.ly/dtpersonal YouTube Tutorials: bit.ly/dttutorials Contact Me: agrabner@dynatrace.com Follow Me: @grabnerandi Read More: blog.dynatrace.com
- 65. Andreas Grabner Dynatrace Developer Advocate @grabnerandi http://blog.dynatrace.com
Editor's Notes
- Yes – I am working for a tool vendor – BUT – you can try this with most of the tools in the APM, Tracing, Diagnostics space out there.
- We all know that since then the world got divided into two big parties. So – just out of curiosity: who is using Apple vs Android? And who is using Windows?
- The phones not only changed the way we communicate – it disrupted many other technologies as well. Look at this! Pope Election in 2005. A SINGLE hand up with an old phone taking a picture
- In 2013 the „picture“ is totally different. The smart phones and tablets not only changing the way we communicate but also disrupting many other things such as taking pictures, getting light when you need one (torchlight), measuring your steps, ...
- So – just to see the difference again in one picture!
- The world is getting even more „disrupted“ – this is a picture from Paris last year when Uber entered the Market. A taxi driver probably thinking: „These geeks from the Silicon Valley – dont even have a drivers license – but are kicking us out of our jobs“
- This is Airbnb making a change in accomodation business. Providing „cheap“ accommodations for those that travel but kicking out folks that used to live in SF all of their live.
- Uber, one of the many companies that are disrupting existing markets with new ideas, new approaches, no fear, no „corporate baggage“, no technical debt and no business debt
- But not only does it disrupt markets, it also disrupts our behavior as human being
- We are all part of it. When you remember how and when we are now interacting with services, companies, products, ... – we consume these services all the time!
- We all carry multiple devices http://www.slideshare.net/sophossecurity/device-infographic-slideshow
- And we are using multi device when we access services http://www.futuristgerd.com/multi-device-path-purchase-commerce-google/
- He starts his book with some very insteresting facts!
- http://www.i-scoop.eu/internet-of-things/
- Insurance companies are looking into smart devices to e.g: find out whether a washing machine is leaking. So – before the basement gets flooded it can notify you. With that you also lower your premium
- But there is more that is disrupting ... There are more people livign within that circle than outside!
- 5 Billion people in the emerging markets. Will „old fashioned“ corporations with their business models tap into this market? Or is it going to be the Ubers of the world? Or is it going to be the new entrepreneurs that will just have the next better idea and build better apps? Credits to https://brucelawson.github.io/talks/2015/velocity/?full#1
- We know that everyone these days can build a globally successful business with just an idea and a laptop. Such as Mark Z or the two tech brothers (12 & 14) founders of GoDimensions. They develop 12 Apps so far with more than 35k downloads. Their idol? Steve Jobs!
- Just to get the basics covered: I hope everyone has their own definition of DevOps by now. It is a lot of things depending on whom to list to our which blogs you follow.
- This was my own initial definition of DevOps when I first heard about the term and I did some reading on why people are that excited about it DevOps I think allows us to scale what works well in a StartUp into larger organizations!
- Synthetic Availability Monitoring -> Clearly something went wrong
- Even if the deployment seemed good because all features work and response time is the same as before. If your resource consumption goes up like this the deployment is NOT GOOD. As you are now paying a lot of money for that extra compute power
- Got a marketing campaign? If you roll it out do it smart: Start with a small number – monitor user behavior – fix errors if there are any before rolling out the rest of the campaign
- In case you are a “DevOps Virgin” I definitely recommend checking out The Phoenix Project (the DevOps Bible) and Continuous Delivery (which is what we actually all want to achieve): Deliverying software faster with great quality and without all potential mistakes that a manual and rigid process brings with it
- They really follow the stories of the first generation Unicorn Companies
- Several companies changed their way they develop and deploy software over the years. Here are some examples (numbers from 2011 – 2014) Cars: from 2 deployments to 700 Flicks: 10+ per Day Etsy: lets every new employee on their first day of employment make a code change and push it through the pipeline in production: THAT’S the right approach towards required culture change Amazon: every 11.6s Remember: these are very small changes – which is also a key goal of continuous delivery. The smaller the change the easier it is to deploy, the less risk it has, the easier it is to test and the easier is it to take it out in case it has a problem.
- But it is not only about delivering features faster – it is also about delivering fast features!
- What happened since then?
- But don’t make the mistake to blindly follow every unicorn out there Taken from http://www.hostingadvice.com/blog/cloud-66-devops-as-a-service/
- Its not about giving Devs Direct Access to Ops Deployments
- The problem is though – when you blindly copy what you read you may end up with a very ugly copy of a Unicorn. Its not about copying everything or thinking that you have to release as frequently as the Unicorns. It is about changing and adapting a lot of their best practices but doing it in a way that makes sense to you. For you it might be enough to release once a month or once week.
- Because this is what might happen: If „Being DevOps“ just means you just increase the number of deployments then you are bound to fail. Here is an example of a bad web application. When deploying this more frequently you will end up in more war rooms
- Another example from a SharePoint app that allows production deployments by SharePoint Admins. A simply change directly in production can have very negative impacts, e.g: deploying a new WebPart with a Data-Driven Performance Hotspot
- Let me give you one additional example of a company that became really successful with their online service
- Let me give you one additional example of a company that became really successful with their online service
- They had a monolithic app that couldnt scale endlessly. Their popularity caused them to think about re-architecture and allowing developers to make faster changes to their code. The were moving towards a Service Approach
- Separating frontend logic from backend (search service). The idea was to also host these services potentially in the public cloud (frontend) and in a dynamic virtual enviornment (backend) to be able to scale better globally
- The Backend Search Service Team did a lot of testing on their backend services. Scaling up and down on demand. All looked pretty good! They gave it a Thumbs Up!
- On Go Live Date with the new architecture everything looked good at 7AM where not many folks were yet online!
- By noon – when the real traffic started to come in the picture was completely different. User Experience across the globe was bad. Response Time jumped from 2.5 to 25s and bounce rate trippled from 20% to 60%
- The backend service itself was well tested. The problem was that they never looked at what happens under load „end-to-end“. Turned out that the frontend had direct access to the database to execute the initial query when somebody executed a search. The returned list of search result IDs was then iterated over in a loop. For every element a „Micro“ Service call was made to the backend which resulted in 33! Service Invokations for this particular use case where the search result returned 33 items. Lots of wasted traffic and resources as these Key Architectural Metrics show us
- They fixed the problem by understanding the end-to-end use cases and then defined backend service APIs that provided the data they really needed by the frontend. This reduced roundtrips, elimiated the architectural regression and improved performance and scalability
- Lessons Learned!
- If we monitor these key metrics in dev and in ops we can make much better decisions on which builds to deploy We immediately detect bad changes and fix them. We will stop builds from making it into Production in case these metrics tell us that something is wrong. We can also take features out that nobody uses if we have usage insights for our services. Like in this case we monitor % of Visitors using a certain feature. If a feature is never used – even when we spent time to improve performance – it is about time to take this feature out. This removes code that nobody needs and therefore reduces technical debt: less code to maintain – less tests to maintain – less bugs in the system!
- And this is how it looks like with Dynatrace AppMon Test Automation Feature. We automatically montior every single test execution in your CI and analyze these metrics per Test and per Build. We automatically detect regressions as every metrics per Test will be baselined. This allows us to STOP A BUILD before it moves to other phases in the pipeline
- In Production we monitor the same metrics for our services. Seeing if a recent deployment had any change in # of SQL calls for a particular feature or the # of internal Service Calls. Helps us to make sure that we do not make bad deployments – or at least be aware of it right away to take countermeasures, e.g: rollback or fix
- If we do all that we can build a pipeline that gets rid of bad code and architectural changes right away!
- With that we can make our users happy 24/7 – at any load