Download to read offline
![MONITORING FEEDBACK LOOP
MaaS Team [24/7/365]
Product/TeamReactive
Proactive
Monitoring points (buttons)
Troubleshooters/HowTo’s
Escalation Path
Trainings
First aid based on Howto's
Fake alarms, lack of
alarms](https://image.slidesharecdn.com/jerzygulczynski-150602131348-lva1-app6892/75/Atmosphere-Conference-2015-DevOps-sum-is-greater-than-its-parts-42-2048.jpg)
Uploaded byPROIDEA
Atmosphere Conference 2015: DevOps sum is greater than its parts
The document discusses the evolution and implementation of DevOps practices, highlighting the transition towards agile methodologies and the benefits of self-organized teams. It emphasizes the importance of continuous deployment, monitoring, and a culture of transparency and collaboration within DevOps teams. Additionally, it presents statistics indicating a growing trend in organizations adopting these practices to improve efficiency and reduce failures.
More Related Content
![[CONFidence 2016] Glenn ten Cate - OWASP-SKF Making the web secure by design,...](https://cdn.slidesharecdn.com/ss_thumbnails/glentencate-owasp-skfmakingthewebsecurebydesignbecomeempoweredbetheneo-160603100034-thumbnail.jpg?width=640&height=640&fit=bounds)
![[4developers] - ScalaJS – web without pain of JavaScript](https://cdn.slidesharecdn.com/ss_thumbnails/scalajs-160531133851-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Atmosphere Conference 2015: DevOps sum is greater than its parts
![Agility and Control from AWS [FutureStack16]](https://cdn.slidesharecdn.com/ss_thumbnails/agilityandcontroldougleeaws-161201210211-thumbnail.jpg?width=640&height=640&fit=bounds)
![Number_Guessing_Game_Dsbsbssbzboc[1].pptx](https://cdn.slidesharecdn.com/ss_thumbnails/numberguessinggamedoc1-251206215042-a076fc05-thumbnail.jpg?width=640&height=640&fit=bounds)
Atmosphere Conference 2015: DevOps sum is greater than its parts
- 1. DEVOPS SUM ISGREATER THAN ITS PARTS Jerzy Gulczyński
- 4. IT WAS YEAR2006 To organise, make it „professional” and the results predictible
- 5.
- 6.
- 7. 50 Envelopes, brochures,stamps, adress stickers There are two ways: 1. One envelope by one 2. „Organized” way Which one would be quicker ? QUIZ TIME
- 8.
- 9. http://dev2ops.org/2012/03/devops-lessons-from-lean-small-batches-improve-flow/ LARGE BATCH STAGE 1STAGE 2 STAGE N SMALL BATCH MASSIVE PRODUCTION VS ONE PIECE FLOW
- 10.
- 11. ROAD TO AGILE In2011 we went into AGILE
- 12. And into DEVOPSin 2012 ROAD TO AGILE
- 13. IS DEVOPS LIKETEENAGE SEX…? www.inzynierem.pl Everyone talks about it No one really knows how to do it Everyone thinks Everyone else is doing it So everyone claims they are doing it.
- 14. REALITY BASED ONPUPPET LAB 2014 REPORT „Our analysis makes it clear that DevOps teams are a growing trend.” 92% of respondents working in DevOps departments are implementing DevOps practices, or already have. DevOps 57% No DevOps 43% 57% are implementing, or have already implemented DevOps practices 16% of respondents were part of a DevOps department Other 84% 16% Named DevOps Departments https://puppetlabs.com/2014-devops-report
- 15. WHAT IS DEVOPS ALLIN A SELF-ORGANIZED TEAM AND ONE CYCLE AGILE EXTENSION CULTURE OF CONTINUOUS IMPROVEMENT
- 16. WHY #1 OF BUSSINES Puppets Labsfindings: OF BUSINESS
- 17. WHY #1 OF BUSSINES „High-performing organizations are deployingcode 30 times more frequently, with 50% fewer failures than their lower performing counterparts.” Puppets Labs findings: OF BUSINESS
- 18. WHY #1.5 JOB SATISFACTION HAPPYCOWS MAKE BETTER CHEESE
- 19.
- 20. HIGH ALIGNMENT ANDAUTONOMY http://blog.crisp.se/author/henrikkniberg
- 21. BLAMELESS POSTMORTEMS WHAT IFI TOLD YOU IT WAS A BUG IN A THIRD-PARTY SYSTEM • Do not blame for failures • Be focused on conclusions and on actions after • Be transparent to Business
- 22. DEVOPS TEAM • Productoriented • Cross-functional (dev&ops) • 4-8 people • Sitting in the same room They Make It They Break It, They Fix It.
- 23. THE TEAM Video Platform IncludingPlayer Team VOD Service Team
- 24.
- 25.
- 26.
- 27. TO BE INFRASTRUCTUREINDEPENDENT Amazon EC2 Amazon S3 Onet DataCenter Credit Card Amazon ELB Business continuity plan 4h a day 1% of traffic Lower costs in the future
- 28. LONG SERVER UPTIME- BAD Server uptime is no longer reason to be proud of
- 29. Chaos Monkey isyour friend Adrian Cockcroft from Netflix said, “Do painful things more frequently, so you can make it less painful” Average lifetime of our Cloud's VMs is 4h SERVER UPTIME SHOULD BE ULTRA SHORT
- 30. DEPLOY OFTEN Number ofchanges on production > 200 daily
- 31. CONTINUOUS DEPLOYMENT Automated andboring process Development Unit Tests, Lint Score Code Review Integration Tests Functional Tests Deployment Monitoring
- 32. CONTINUOUS DEPLOYMENT Create across-organization CD Team
- 33.
- 34.
- 35. CONTINUOUS DEPLOYMENT Provide thedocumentation
- 36. CONTINUOUS DEPLOYMENT Measure maturity& help achieve higher level
- 37. AB TESTING WHEN DEPLOYING AB TESTINGWHEN DEPLOYING Deploy on 10% of traffic, check, increase
- 38. PROVIDE LOGS &EVENTS • Log everything • Provide logs and events open for all devops we collect 130k logs, events per second 1 mln metrics in Graphite Online console, Graphite, Hadoop
- 39. Monitoring is intendedfor administrators only. People who don't know how to create it and thereby who are not able to interpret it are kindly REQUESTED NOT to run alarms and NOT to disturb administrators. MONITORING THE PAST
- 40. PROACTIVE MONITORING Monitoring TVsin Teams’ Rooms
- 41. Monitoring Center andMaaS Team
- 42. MONITORING FEEDBACK LOOP MaaSTeam [24/7/365] Product/TeamReactive Proactive Monitoring points (buttons) Troubleshooters/HowTo’s Escalation Path Trainings First aid based on Howto's Fake alarms, lack of alarms
- 43. DON’T JUMP INTODEVOPS IN A WATERFALL WAY
- 45. LINKS One piece flow:https://www.youtube.com/watch?v=Dr67i5SdXiM Puppet Labs Report: https://puppetlabs.com/2014-devops-report Microservices: http://martinfowler.com/articles/microservices.html Small Batch: http://dev2ops.org/2012/03/devops-lessons-from-lean-small-batches- improve-flow/ Pets vs Cattle: http://www.lauradhamilton.com/servers-pets-versus- cattle
- 46.
Editor's Notes
- #2 Hello everybody Thank you for the invitation. It is great to be here and great to have the oportunity to share my experiences with you guys. My name is Jerzy Gulczyński I’m from DreamLab I’m Director of IT Platforms Dev&Ops and DreamLab Board Member Today i’d like to tall you about the idea of devops ; why we implemented devops and what we did, what is important ! Let me start from the beginieng.
- #3 We are a part of big media grup – RAS that consists of RAS Poland, Onet, NK, Skąpiec & Opineo, and also RAS Serbia, Hungary, Slovakia with headquarters in Switzerland.
- #4 DreamLab is Research Development & Operations Company and we build and operate a lot of services like Onet.pl biggest horizontal portal in Poland, Onet Email, Vod.pl, zumi.pl, nk.pl, skapiec, sympatia.pl, and RASP Services newsweek.pl, przegladsportowy.pl, forbes.pl and many other and we also prepare our technology to be multenant and multilanguage. and to launch services in Hungary and Serbia We create portals for 16 milion users monthly, we have huge video traffic also. We utilize one is of most modern DataCenter in Europe Our team is about 300 engineers working in 40 Product Teams located in Krakow, Warsaw and Wrocław
- #5 When we started in 2006 as a separate R&D center for Onet and at that time also for TVN, we weren’t as mature as we are today. The only thing we had was chaos and too many things to do at the same time, we had too many open projects, too many customer at the same time and we weren't able to manage them Thats why we decided to ogranise and to make it proffesional. Let’s quickly go trough the first chapter of Dreamlab that is already behind us. I hope this will help you avoid mistakes that we made.
- #6 We implemented Havy methodologies and silo structures We appointed the central project management office We divided teams into competence orinted functional siloses We implemented a waterfall process based on PMI’s Project Management Body of Knowledge and OpenUP Framework. This is one side of the story. On the other side we had operations - with 24/7 1st line of support administrators, we also had Network, Unix, Dba siloed teams there. And we implemented ITIL. We were even awarded in 2008 for the best ITIL implementation in Poland by Help Desk Institute.
- #7 Development was driven by change and iron triangle (cost, time, scope commitmens) and operations by the uptime ofcourse Project teams were appointed dynamically. We took for example 3 persons from frontend, 2 persons from midleware and databases and we had a project team assigned for some period. When we finished development in the project - we took our software to the wall that separates development from operations, and we threw it over the wall to forget about it and to let administrators worry about it.
- #8 Before I go to the lessons learned a neet to ask you one question. Let’s say you have to send brochures to your clients. You have 10 envelopes, brochures, stamps, adress stickers. There are two ways of preparing it. First. One by One. You take one envelope, you stick a stamp to it, you stick the adress sticker, you fold the brochure, you put it into the envelope, you seal the envelope //and then you take another envelope and you repeat all operations for next envelopes. There is another mehod - you can do it in let's say an organized way, you stick every stamp to each envelope…. Who of you is choosing method one - show of hands please The Second method ? – show of hands please Most of us feel that the second method is quicker but Unfortunately it isn’t Our intuition suggests us that we'll be more effective and we’ll be getting better at it – by repeating the same action many times and we forget that we lose a lot of time for moving and ordering large batches of envelopes that are not yet ready. I know - It is hard to beleive. It was proven in many experiments ; one of which was even recorded and published on youtube. I’ll provide you a link this video at the end of presentation. -- One piece flow vs batch production 2:56 -> 3:42
- #9 Why first method is more affective ? The First method is an example of one piece flow. Its good performance stems from the surprising potential of small batches. Advocates of lean manufacturing discovered potencial of small batches dozens of years ago. (dazyns)
- #10 What is a batch ? Size of batch is the unit of work that is passed from one stage to the next stage in a process. For example you stick 10 stumps to 10 evelopes and than you pass whole result to the next stage. The problem of large batch is that - it is not optimized for effectiveness of the whole process. There is another problem of large batch – even more important - late feedback. Let's imagine you chose the second way – the one based on large batch - and in the last phase you discovered that there was a defect in each envelope - you couldn't seal the envelope – there’s no glue. You have to upack everything, buy new envelopes and do it again. Using the first method you would know about it at the very beginning and you have complete product quickly The smaller batch leads to quicker feedback, less complexity, less communication problems, less risk of regression and errors Agile & DevOps is a about small batches, quick results and quick feedback Waterfall is about large batches ; clients has got large batch of requirements ; UX designs large project ; and so on. Large batches are also handed over between siloed teams – backend, frontend
- #11 Go to our lessons learned. We ware really mature, ; we were finishing more than 80% of projects on time, in budget, in scope. But quality was low and technical dept was significant. We were slow and ineffective. A Team that uses Waterfall and Siloses is like a cheetah closed in a car. It feels it can run faster but can not. Large batches were everywhere. Client has huge … and count on such reaction. It's risky. Such bureaucratic culture is destructive to performance, job satisfaction and also creativity. Employees in such an environemnt feel like in a factory, like cogs in a machine. They can’t be creative. They become frustrated. Relationsip – IT – bussinnes is also poor. It’s a Demand – Supply relation. /// than common work to common goal. We drew conclusions (DRU) and we decided to leave this world of pain. ///Unfortunately iron triangle has substantial impact on the quality and technical debt. ///They feel disempowered, they think they have no influence on the product and process, they don’t see results of their job quickly because the whole project is so long, and tyhe become frustrated ///Job satisfaction and engagement was really low. ////They wait for another team, they wait for deployment approval, they loose their time on misunderstandings, communication problems, they don’t see results of their job quickly because the whole project is so long. ////Ofcourse some projects were cancelled because they took to long and business condition has changed. ////Sorry, you assumed that all of these features are needed by your client. You didn’t want to build MVP or to add features in the interation to get quick feedback.
- #12 and we released the cheetah from the cage in 2011
- #13 And more then year later we had two cheetahs running alongside– devs and ops.
- #14 What do you think – is devops like teenage sex ? … It’s not clear to me how it is in Poland.
- #15 Let’s look at devops globally Puppet labs did a survay and provided intresting results in State of DevOps Report 2014. ///It was their second report (the first was published in 2013). More than 9,200 people from 110 countries responded to their survey including Europe (25.4% were from Europe). (point) They found that 57% are implementing or have already implemented devops practices in their companies What is also intresting - 16% of respondends are part on named devops departments and 92% of them are implementing devops practices or have already done it. They also found that devops is on a growing trend. //Respondends were from variuos business areas, from really small, medium and large companies. ////The majority of survey respondets work in the technology and web software industries (23 percent and 11 percent, respectively), but there were also from Education (8 percent), banking and finance (7 percent) and entertainment and media (7 percent) and other. Organizations of all sizes: from tiny startups to 10,000-employee companies //////, from shops with fewer than 100 servers to large enterprises with more than 10,000 servers under management.
- #16 What is DevOps In my opinon DevOps is an extension of Agile. It’s like agile on Steroids because thanks to devops you can be even faster For me It's hard to imagine DevOps without Agile and vice versa. This is software development, operations and quality done in a one cross-functional and selforganized team and in one cycle which is based on just small batches. This is a team where developers think more like ops and vice versa. Devops is also a Culture where you should fail quickly in order to learn and improve quickly DevOps is also collection of some practices ///It’s about trust, autonomy, selorganizations.
- #17 So Why Implement devops. Before we start implementing devops we should answer this question to ourselves – WHY We should also have answer at this question to your organization and your team. Why to do it ? Becaouse it’s trendy, becaouse everyone claims they are doing it ? This is not the reason. The first and most important are business reasons. ///To your organization and to your team you should clarify the reasons WHY you want to implement devops
- #18 Because of higher effectiveness, Faster T2M, Fast Feedback, Better and more innovative product, Better Quality, Better Uptime, Less cost of infrastructure, From Puppets Labs report. Puppet lubs found that company who have implemented devops are far more eficient then those who have not. They also found – that is very interesting - that high perpofming organizations are doploying code 30 times more frequently with 50% fewer failures then their lower performing counterparts. Today market is very competitive, clients are very demanding and the way you do your software is in my opinion crucial.
- #19 The second reason is a job satisfaction Someone said that „happy cows make better cheese” It also works for as - people When we are happy we are more engaged, more effective, we are more creative – and it also leads to better business outcomes Why we are happy / satisfied in a devops Because of High-trust culture, fast results, climate of learning, good realationship between ops and dev, because of automation thanks to which people find time for creative and of proactive work instead of doing repetitive one, high autonomy, selforganization Ok so what you shoud do to implement devops, what is important.
- #20 You should take care of the culture. DevOps is first and foremost about the culture - not only about tools and practices. It’s easier to implement automation, infrastrucure as a code then to change the company culture. Devops is about high Trust that is a driver for good information flow, good collaboration, learning from failures, continous improvment and new ideas Sef-organization of the team is related to high autonomy and responsibilty - it is opposite of Survival mode where team doesn’t have time for anything, fights with fires and can’t do anything without their team leader. The way of the team from sirvival throgh lerning phase to selforganizied is well discribed in ROY OSHEROV book – Notes to a software team leadr. ROY decribes how you shoud adopt your team leader style and how to chage your team. Devops culture is about Autonomy, Mastery and Purpose which are strong drivers of engagment. This is a Daniel Pink motivation idea that is called Drive.
- #21 This is Henrik Kniberg slide – who is a consultant working in a Swedish company I like this slide very much because it describes alignment and autonomy perfectly. High alignement and high autonomy is what you should achieve in agile/devops culture. Define clear goal – problem, give them also wider perspective (why you wat to do it) but don't tell the people how to solve the problem. Don’t tell them Build the Bridge. Allow them to find the best solution of the river problem ; mayby a bridge is not a good solution ; mayby they will build a ferry or the boat.
- #22 Allow them to create company culture and to realize their own product ideas. These are pictures from one of our 24 hackday which was organized only by our people – by the volunteers ///HackDay is an event when come together and work for 24 hours non stop to create a prototype of their own idea.
- #23 Allow them to share their knowledge under on external and internal events and to feel the mastery Onet Barcamps are also organized by dreamlab volunteers
- #24 If you are devops and you take care about the product you should have postmortems after failures. The goal of blameless postmortems is to draw conclusions and to improve product and process. It also builds a trust. Do not blame anybody and do not punish for failures Treat every failure as a lesson. Otherwise your coleges will be closed, problems will be swept under the carpet and you will kill collaboration. Never hide failures from your business partners or clients Tell them what happed, what was the imact, what you will do to prevent from such failure in the future We send information about every failer to business and IT – we text it and also send via email ; and we provide postmortems after failure to them.
- #25 Team should be product oriented and they should love their product. They shoud create a business value. They souldn’t be beckend or frontend oriented. Backend is not a product and also it’s not a business value. Team should be cross-functional to be able to develop and also to operate the whole product by themselves. When they break it they should fix it. for best collaboration and effectivnes there should be 4-8 persons - also should be sitting in the same room Productivity for such small grup is even 30-50% higher then for gropus over 10. And event 100% increase in Productivity if team sits in the same, closed room „When we are woken up at 3 a.m., we take care about the product, we have a focus on quality when we write code, we don't deploy before they leave the office or before long weekend, we do continuous integration, monitoring and troubleshooting instruction for 1LineOfSupport etc.” //In such cross-functional environment they will be also more creative //Team should create working product and business value. //There is more likelihood of generating breakthrough solutions //Does it mean you have to have regular administrators there ? – it depends on how your team is isolated from the infrastructure/iaas ; for example – if your team uses classic architecture – LAMP they should have administrators //Please also note that a cross-functional team will be more creative because they will join knowledge and experience from different areas.
- #26 What if your product is too large or too coplicated to be developed in a one 8 persons team. Divide product vertically ; for example video player that is a complicated solution should be assigned to video platform team as a part of video platform The Rest of the service like the film catalog and any other functions to another team. The key is that the team is responsible for whole functionality from top to bottom Complexty of such integration is simple. Player is just embeded on the page ; Try compare it to backen – frontend integration. In such solution - Video platform team feels responsible for the final result, they create business value, they provide player directly to user – not only the backend of the video platform.
- #27 In old-fashioned infrastructure that was common to treat servers as pets and to give them also cute names. Every server was manually and individually configured and tuned.
- #28 Today, Instead of treating machines as pets we treat them just as cattle. We are like cowboys. And we don't cry if any one of them is sick or even dead because our services are stil healthy. it is not possible to login on server and to change cofiguration manually.
- #29 In last few year we completely changed our technology stack. We use asynchronous (ejsynkronys) technologies like python tornado and node.js. We implemented PaaS platforms and they are used by devops - like RDBS, NOSQL DB, SEARCH, QUEUE, OBJCET STORAGE. Our envirnment is fully automated in the cloud - starting from commodity hardware finishing on applications. PaaS orchestrator takes information from scorbord and it just knows it has to run a new instance of an application. It asks IaaS for some type of virtual machine. When it is ready, application is downloaded using bittorent network from repository of packages, installed and run on it. Thanks to it the whole envirment and at the end also Onet can be run even from the scratch - from 0. ///Currently we work also under Software Defined Network for out Cloud Env //We don't waste our time for administration of single servers and we no longer treat servers individually as our pets.
- #30 But thanks to the automation we run our Cloud and Onet also in the Amazon. We utilize two Data Centers in Krakow but only in one we have our full cloud environment. We use Amazon EC2 and S3 as our disaster recovery site in case of failure of our main DC To be all the time prepaed for such scanario we run our cloud in amazon every day for 4 hours for 1% of traffic. Traffic and duration is limited becaouse we don’t want to pay them too much ;) Main instance of cloud is based on our DC, recovery instance is credit card driven infrastructure. In this scenario it is cost effective ; thanks to it we haven't spend a lot of money on hardware in our second DC. We are also sure that in a few years computing power from huge provider will be less expensive than our own DataCenter That’s why we are just ready to swich.
- #31 We are sure that Server uptime is no longer reason to be proud //for us.
- #32 Instead of having long uptime try make it ultra short Thanks to Chaos Monkey Average liftime of our Virtual machines is about 4h. Chaos Monkey just switches off our virtual machnines in the cloud. Applications are going down and then they are automaticly run again somewhere in the cloud. Thanks to it our applications live (lyv) in permanent failure and they are ready for a true failure. The whole environment and orchestrators are constantly tested. /// We will also have chaos monkey in our development environemnt This will convince our developers to commit to source repository very often and to automate everything even in the development environment.
- #33 Do you remember small and large batches ? Smaller change is a smaller complexity and a smaller risk of failure at the end. We have got about 200 deployments – small changes every day on production. Paradoxically many organizations lengthen their release cycles and this approach just increases the pain. Our goal is to make deployment process boring and automatic. To do it you shoud deploly small changes often and you should have -- Continuous deployment process. //Deployment shouldn't be anything unusuall for you in DevOps environemnt. //Microservices as independent small deployable components would be very helpful also. //Paradoxically many organizations lengthen their release cycles and this approach just increases the pain. //You shouldn’t afraid of it.
- #34 This is clear how it works ; so I don't want to describe it in detail. The goal of CD is to make your deployment process boring, repeatible (rypityble) and automated. You can save a lot of time thanks to deployment automation You won’t be wondering how to deploy and what you have forgoten when you deploy. So how to have CD ? In the past we tried to do it many times with poor results. Every team did it on their own Two years ago we decided to change it. We also realized that we had to spend money and time on it. We defined clear goal and clear vision of continous deplyment.
- #35 We Engaged the people. We appointed cross-organization CD team, so we asked volunteers to join the team. They prepared the first version of environment, tools and the process and still today they are taking care of the process and tools. They are ambassadors, evangelist and users of this process and they help others implement CD.
- #36 They chose Atlassian: Jira + Stash + Bamb, Sonar Cube to manage code quality and selenium for functional testing. the first three tools are perfectly integrated which makes this environment very friendly for teams ; ///last two are possible to integrate. These tools cover whole workflow – starting from task in backlog through tests, code review to automatic deployment ////– thanks to deployments profiles. /// Plugin for sonar cube For python – code standards ; double variable ; best ; pep8 ; some antipaterns
- #37 Our team configured whole environment, they provided standards, guideline, templetes, quality profiles, they defined workflows etc. to make CD easy for everybody
- #38 Thye prepared a documentation. To tell everybody how to join, how to use it and so on.
- #39 I also suggest you to measure the progress and maturity of continous deployment within organization. To measure, identify problems and to help everybody improve. You can check many things like: How many project are moved from the old source repository to the new Do teams deploy on production whit codereview What is test coverage And so on.
- #40 This is a another practice we utilize it widely at DreamLab. To minimase risk of failure you can also deploy on some small percentage of traffic. This a post deployment report for video player which is based on Real User Monitoring. We deploy new version of our player on some small part of traffic eg. for 10% of users, than we check the results and if we see we are safe then we can increase the number of users for whom we deploy As you see we are able to compare Player Init time and many other parameters like Bitrate, Buffering for two verions of our video player - version 70, and a new one 72 //- this has just been deployed. ///We measure and collect every event that happened on a client site in a every browser in video player real time.
- #41 Logs and events are crucial for devops - online and offline data This is also crucial in a cloud environment. Thanks to it you can have proactive monitoring, discover alarming trends, diagnose and solve problems fast, you can deploy often. We collect about 130k logs per second from every layer in our cloud into a central place that is open for developers. Events - Monitoring Metrics are in Graphite, online logs are available via console ; history logs are in Hadoop. We don’t focus on system and applications metrcs only ; We measure User Expericence thanks to RUM – we measure times and errors on a client site from on browser in realtime, video player events and so on. ///This way you have all these data in one place. ///because your server and your application is somewhere in the cloud so you can’t login to it.
- #42 This is the screen from our old monitoring system and this information is real - it’s not a fake or joke. So as you see in tha past monitoring tools were reserved for administrators only. This also clearly shows how high the wall of confusion was.
- #43 In devops environment proactive monitoring is a must. You are responsible for the product – for development and operations. You have to know what's happening in your product ; currently, also in short and long perpective - to take actions if required. to discover alarming trends, diagnose and solve problems fast To have proactive monitoring you mast have tools ; We use many tools – the most important is graphite, reworked tattle and splunk. You should also have monitoring tv’s in every room for every team. And this is a very good pracise. Thank to it teams really take care of about the product and product monitoring on a daily basis and they develop monitoring in parallel to product development. On the left you see Continous integration status, on the right product monitioring. //what state your product is in
- #44 And we also have Monitoring Center and Monitoring as a service Team active 24/7/365 – in our Data Center.
- #45 There is a strong correlation between monitoring in DC (image on the left) and monitoring of each product in a team rooms (on the right) Every product or platform has it’s own alarm button on screen in DataCenter – for example vod product or email product or cloud Such single button on DC Screen is related to the whole screen of buttons in the team’s room. If any of product’s buttons is going red then the related button on the DC Screen is also red. Each Product team provides … to MaaS team. They work proactively. MaaS team ensures fast first aid for every product based on Howtos and their knowledge. But, they also identify fake alarms, lack of alarms and they do reports for product teams. Product Teams based on this feedback can improve their monitoring and product. They work together in a continuous feedback loop. //Main part of their job is reactive.
- #46 Look at your data it in shot and long perspective also to find alarming trends. Do not focus on system's and application's metrics only. Collect also UX metrics. This gives you information how your product works on the client side. For example Buffering Ratio is a percentage of time when video player buffers and interrupts watching. If this parameter exceeds 2% than watching time decreases dramatically which affects the business
- #47 Another example . We measure page speed on every single page in every browser that visits our sites in real time. As you know page speed is important UX predictor and it clearly correlates with business results. We have got alerts on our monitoring system to be notified when time thresholds are exceeded. But also we look from wider perspective provided every week in special speed report. As you see this peak was an impuls for our team to optimize. This allows us to discover alarming trends and take an action quickly.
- #48 If you really want to implement devops. Don’t jump into devops in a waterfall way ; Like this guy to a frozen swimming pool. Build clear a vision and goals for your team and your organization. Engage your team into this change ; you can't do anything without them Implement it in the iteration ; for example take one team, one client, one product, prove that it works, build a success story. Acquire ambassadors. Buid a critical mass then things will just happen by themselves. Take an external coach who has authority, knowlede and practise ; a person who give you some practical clues and help you Based on my experience I can tell you that IT is usually the leader of change and I know that it is possible. As IT we changed the way of product development in the whole organisation. We changed also the way our business plans their project Agenda. We do it together in quarterly cycles. I honestly wish you good lack and I hope you will find the curiosity and determination to do it.
- #49 Thank you for your attention
- #50 There are some helpfull links
- #51 And also books