5 Practices for Better, Cheaper, Faster Service Delivery
•
2 likes•74 views
From DockerCon 2018: 5 ideas behind LendingClub's DevOps technology transformation from 2012 to 2018.
Recommended
More Related Content
What's hot
What's hot (15)
Similar to 5 Practices for Better, Cheaper, Faster Service Delivery
Similar to 5 Practices for Better, Cheaper, Faster Service Delivery (20)
Recently uploaded
Recently uploaded (20)
5 Practices for Better, Cheaper, Faster Service Delivery
- 1. v Rob Schoening 5 Practices for Better, Cheaper, Faster Service Delivery VP, TECHNICAL OPERATIONS, LENDINGCLUB Ashley Sun SOFTWARE ENGINEER, LENDINGCLUB @ashleycsun
- 3. Who Are We?
- 6. Slide 6: I dragged the photo to be longer because we would like to cut the black horizontal bar with all that text out of the slide.
- 7. DevOps @
- 9. 2013 2018
- 10. 2013 2018 5 500 Microservices
- 11. 2013 2018 5 500 Microservices Datacenter Cloud
- 12. 2013 2018 5 500 Microservices Datacenter Cloud VMs Containers
- 13. 2013 2018 5 500 Microservices Datacenter Cloud VMs Containers $1B $35B
- 15. # of microservices over time 2013 2014 2015 2016 2017 2018
- 16. # of DevOps engineers over time 7
- 17. What is our job?
- 18. To make computers do things… What is our job?
- 19. To make computers do things… ...so that people can do things…What is our job?
- 20. To make computers do things… ...so that people can do things… ...better, faster, and more efficiently… What is our job?
- 21. To make computers do things… ...so that people can do things… ...better, faster, and more efficiently… …And have more fun! What is our job?
- 22. How?
- 23. 5 Practices
- 24. 1
- 25. Embrace the architecture you really have … … Not the one you think you have… …Or the one you want to have.
- 26. What does your world look like?
- 27. 27
- 28. 28
- 32. Neo4jNeo4j Mercator Monitoring Alerting Reports Real-time Adhoc Querying Automation Load Balancer App Instance Health Check
- 33. Neo4jNeo4j Mercator Monitoring Alerting Reports Real-time Adhoc Querying Automation Automation Deployment What pool to deploy to? Are multiple revisions deployed to a single pool? Are any of our pools degraded or down? Do we have a Single Point of Failure? Load Balancer App Instance Health Check
- 34. 34
- 39. Neo4jNeo4j Mercator Monitoring Alerting Reports Real-time Adhoc Querying Automation Automation Deployment What pool to deploy to? Are multiple revisions deployed to a single pool? Are any of our pools degraded or down? Do we have a Single Point of Failure?
- 40. Neo4jNeo4j Mercator Monitoring Alerting Reports Real-time Adhoc Querying Automation Automation Deployment What pool to deploy to? Are multiple revisions deployed to a single pool? Are any of our pools degraded or down? Do we have a Single Point of Failure?
- 41. So Where Do I Invest? Model your infrastructure services Model the relationships between your infrastructure services Build meaningful automation around this data
- 42. 2
- 43. Embrace What You Hate Most: Make It Awesome Source: Do the Right Thing (1989)
- 44. “I Hate Ops!” - Me, VP Technical Operations
- 45. Things Developers Hate On-Call Security Remediation Audit / Compliance Outage Lost Revenue etc.
- 48. So Where Do I Invest? Minimize On-Call Pages Secure-by-default Compliant-by-default Self-Healing Architecture Partial Degradation etc.
- 49. 3
- 52. PROBLEM In 2013 when Docker barely existed we asked: How do we embrace containers without making massive infrastructure changes?
- 53. Necessity is the mother of invention!
- 54. Infrastructure-as-Code as a constraint… ...forces problems into three categories: … and eliminates “human error” DESIGN DEFECTS ENHANCEMENTBUGS
- 55. From: “Can’t we just…” To: “The system doesn’t currently do that…”
- 56. From: “Why didn’t you…?” To: “Why didn’t the system…?”
- 57. Embrace Docker It provides constraints!
- 58. So Where Do I Invest? Find a balance of freedom and responsibility Embrace constraints to lower cognitive load on service owners
- 59. 4
- 60. Convexity
- 61. Convexity Success At Scale Invest in things where payoff, benefit and value increase with scale and complexity Be antifragile
- 64. Example Monitoring application threads for anomalous behavior
- 66. Problem: Blocked Threads Difficult or impossible to reproduce outside of production Time-consuming to diagnose Evidence disappears
- 67. Solution Collect thread dumps from all running microservices every few minutes. Search for and identify anomalous behavior and pro- actively notify service owners through chat.
- 68. Sample Alert
- 69. How Is This Antifragile? • Value increases with the number of services in use • Faults drive betterment • System pre-emptively notifies service owners of problems that they didn’t even know existed • Problems are transformed into knowledge and human understanding of the system • The more services screw up the more we learn and benefit
- 70. Where Do I Invest? • Upside benefit should increase with scale • Upside benefit ideally unbounded • Downside risk does not increase with scale • Downside risk ideally bounded • Embrace loss aversion – use human pain as motivation • Minimize cognitive load
- 71. 5
- 72. People Say Ignore What at Conferences Works For You Do What
- 73. Complex Solutions for Simple Problems
- 75. Creativity from constraints 5 Practices (for better, cheaper, faster service delivery) Embrace the architecture you have Embrace what you hate most– Make it awesome Convexity: Invest in things where value increases with scale Don’t listen to people at conferences; do what works for you
- 76. For More… OSS MERCATOR: https://github.com/LendingClub/mercator Get @ us! @ashleycsun
- 77. We’re hiring! rschoening@lendingclub.com asun@lendingclub.com Join us on our journey! Write awesome software and have fun!
- 78. ?