Cleaning The Part: Reclaim your Logging
•Download as PPTX, PDF•
0 likes•443 views
Monitoring is great. So are logs. But when all your dashboards are red and your inbox is full of alerts, how do you separate the noise from an actual emergency? With a dirty park, it becomes easier for teams to add more garbage to the pile compounding the problem. In this talk I will walk you through the approach I'm using to help my company move toward 0 error logs. Lack of visibility and lack of ownership have left us with millions and millions of error logs monthly most of which are useless noise. I'll take you through some of the technology choices we made to make error logs easier to consume and harder to avoid. I'll also share my learning on how to drive motivation for a project that feels never-ending.
Recommended
More Related Content
What's hot
What's hot (19)
Similar to Cleaning The Part: Reclaim your Logging
Similar to Cleaning The Part: Reclaim your Logging (20)
More from Matthew Campbell, OCT
Recently uploaded
Recently uploaded (20)
Cleaning The Part: Reclaim your Logging
- 1. Cleaning the Park: Reclaim your Logging Matt Campbell matthew.campbell@d2l.com @beardedcoder
- 2. Matt Campbell Engineering Director with D2L Leading project to achieve proper web-scale in AWS Previously lead move to monthly deployments matthew.campbell@d2l.com @beardedcoder
- 3. What can I expect from this talk? • Who is D2L and what do they do? • How we ended up with a dirty park • How we tried (and failed) to clean up • How we tried (and failed less) to clean up • What I’ve learned along the way
- 6. Millions of logins daily at peak TBs of aggregate data PBs of aggregate content Clients with Multi- TB DBs
- 16. Dev adds code Code generates log
- 18. Logging • Structured information written out during operation of code. • Can be at various levels (info, debug, warn, error, fatal). • Typically used to debug exceptional code behavior. For the purposes of this talk, when I refer to logging, I mean Error/Fatal level logging (the really bad stuff)
- 23. Users Application Servers Database New Application Servers Migrated DatabasePartially Migrated Database
- 24. Made visible count of logs made each day
- 26. Users Application Servers Database New Application Servers Migrated Database
- 30. Dev and Ops were very separated at the time Tried to solve logging for all things at once Tried to solve for all regions and all instances at once
- 31. They didn’t have a strong why
- 50. Dev adds code Code generates log Dev reviews log and makes change
- 68. Top Tips
- 71. Go for something sticky
- 73. Everyone has inertia to overcome
- 74. It’s OK to acknowledge this will take awhile
- 75. Celebrate the small victories
- 76. Don’t be afraid to start
- 77. You don’t need to know how to finish
- 78. Cleaning the Park: Reclaim your Logging Matt Campbell matthew.campbell@d2l.com @beardedcoder
Editor's Notes
- https://www.soswildlifecontrol.com/wp-content/uploads/2017/01/qtq80-vB3b6f.jpeg
- Who is D2L? What is it that we do? Image Copyright 2016 D2L Inc.
- We’re global https://upload.wikimedia.org/wikipedia/commons/0/09/BlankMap-World-v2.png
- We’re data intensive <See if you can get stats on logins, amount of data, size of DBs>
- In the early days of D2L, there weren’t fancy technologies like ElasticSearch and fancy third-party logging vendors. So we dumped the logs in the spot we had: SQLServer http://1.bp.blogspot.com/-M1MjcnIA7fU/Voz-EKMq5PI/AAAAAAAAL-w/xvywHnJugm8/s1600/rolling-logs.jpg
- When the company was small this worked just fine. Every had access to everything and people worked close (as in physically) together.
- However, as the company grew, people had to specialize and we had to start restricting access to client data. This meant that developers no longer had easy access to the logs that were being generated https://i.ytimg.com/vi/TeYaQGbD6Xc/maxresdefault.jpg
- However, we knew logging was a good thing so we kept adding logging to the code. For a time I imagine devs still had insight (through back channels), but overtime that would have waned.
- However, we knew logging was a good thing so we kept adding logging to the code. For a time I imagine devs still had insight (through back channels), but overtime that would have waned.
- Eventually, we were adding logging because the code we saw had logging in it. And as much as we don’t want to admit it, we copy other code to help get our work done faster. And in copying that code we don’t also think if what we are copying should all remain. So, more and more logging was added. https://i.redd.it/tjltyl5m332z.jpg
- This meant we had 100,000’s of logs monthly (just at an error level) that were rarely looked at. https://upload.wikimedia.org/wikipedia/commons/c/c7/Logs.jpg
- However, all these logs were consuming disc and IOPS on our databases https://img.gawkerassets.com/img/19aanrziuk65tjpg/original.jpg
- https://www.soswildlifecontrol.com/wp-content/uploads/2017/01/qtq80-vB3b6f.jpeg
- https://www.soswildlifecontrol.com/wp-content/uploads/2017/01/qtq80-vB3b6f.jpeg
- To my knowledge we’ve had 3 previous attempts to correct this problem https://i0.wp.com/mobcblog.wpengine.com/wp-content/uploads/2015/09/52.jpg
- The first attempt spun off of my past project, moving D2L to continuous delivery. My manager at the time (and the architect of the CD solution) realized that the amount of noise in the logs meant we couldn’t use them to figure out if we had broken anything while moving the deployment model forward.
- So we built a widget on a dashboard that showed the number of error logs generated that day.
- And nothing happened. While, something happened, we all felt really bad about how many logs we made.
- We hadn’t fixed the main problems blocking us: Simple access to the logs and an aggregated view across all instances.
- This is when the idea of combining all the logs together into a Centralized solution first came about http://cdn1us.denofgeek.com/sites/denofgeekus/files/2016/10/captain-planet-movie.jpg
- http://teamgaffney.com/wp-content/uploads/2016/02/Return-to-Sender.png
- https://i.imgflip.com/y9sav.jpg
- Focused on solving logging for the sake of logging. But no one cared about logging.
- Full disclosure, we’re not actually done yet <gasp> https://fluffrick.files.wordpress.com/2012/01/richmond.jpg
- So why is this time any different?
- We have a stronger why behind the move. This time it is spurred by our large-scale migration into AWS. https://cdn.vox-cdn.com/thumbor/tlEpA9YH7R4FUtj2Bf347AoeP8I=/0x0:308x164/1200x800/filters:focal(95x65:143x113)/cdn.vox-cdn.com/uploads/chorus_image/image/54087325/images.0.jpg
- Moving into AWS provided us with access to technologies that we didn’t have before
- With our move to AWS we also are rearchitecting our application for multi-tenancy. The way we currently log is a blocker to that.
- All of this gave us a stronger vision for the project than just fix logging.
- https://nbchardballtalk.files.wordpress.com/2017/02/field-of-dreams-e1486999847754.jpg?w=1200
- The goal was to still get to 0 logs. With error logs under control we can actually start to treat new error logs as critical events. However there were far too many logs for one team to handle. Even with the new people who came on board once Kibana was out, it wasn’t enough to start making progress. http://i2.cdn.turner.com/money/dam/assets/150109042203-line-graph-ground-breaking-1024x576.png
- Recall that we want to get to 0 error logs so that any logs can be treated as an Incident
- Enter LogDropper. We started an initiative to
- The emotional cycle of change
- http://www.fitcrazy.tv/wp-content/uploads/2016/09/blog-lesson-from-barbell-main.jpg
- http://static.tabatatimes.com/wp-content/uploads/2013/10/Sisyphus-Image-01C.jpg
- https://greatperformersacademy.com/images/images/Articles_images/small-wins-happiness-success.jpg
- https://upload.wikimedia.org/wikipedia/commons/6/63/African_elephant_warning_raised_trunk.jpg
- https://www.soswildlifecontrol.com/wp-content/uploads/2017/01/qtq80-vB3b6f.jpeg