Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

[KubeCon NA 2018] Effective Kubernetes Develop: Turbocharge Your Dev Loop - Philip Lombardi


Published on

Every software development cycle is rife with inefficiency. Seasoned devs know the pain of getting access to essential remote systems, waiting for tests to run (and then fail), or debugging with only log files. This talk teaches you how to best leverage Kubernetes, remote infrastructure and related tooling to create a dev cycle that maximizes velocity and minimizes developer friction and frustration.

Using tools such as Kubernetes, Docker and Telepresence, I will walk attendees through several advanced techniques that can be used to produce an effective developer experience and optimized dev loop. The goal of this is to eliminate many sources of frustrating inefficiency and reduce cycle time between releases. I will demonstrate how to incrementally adopt some of these techniques and how to approach introducing new and unfamiliar technology and techniques to skeptical dev teams.

Published in: Software
  • Be the first to comment

  • Be the first to like this

[KubeCon NA 2018] Effective Kubernetes Develop: Turbocharge Your Dev Loop - Philip Lombardi

  1. 1. Effective Kubernetes Development Turbocharge Your Dev Loop Philip Lombardi @TheBigLombowski
  2. 2. This Talk ● Share the Cloud Native workflow we have evolved over three years ● A journey about how we got where we are along with some broad stroke technical and communication insights 2@TheBigLombowski
  3. 3. $(whoami) ● Internal dev tooling and infrastructure ● Occasionally hack on product ● Very impatient… on a mission to make Cloud Native development less cumbersome 3@TheBigLombowski
  4. 4. 4 Dev-Test Deploy Build Code Dev-Loop @TheBigLombowski
  5. 5. 5 Dev-Test curl ... e2e “Stuff” Deploy kubectl <...> Data migrations “Stuff” Build Binary Docker Image Assets Code func main() { ... } if __name__ == “main” fun main(args: Array<String>) Dev-Loop @TheBigLombowski
  6. 6. Late 2015 ● Not on Kubernetes yet (kube 1.0 brand new) ● Bootstrapping our backend… need to ship fast and ship often ● Dev loop is very primitive… 6@TheBigLombowski
  7. 7. 7 Dev-Test Wild West Deploy Mix of: Shell, Terraform and Ansible Build Build + Package Binary Build VM image Code fun main(args: Array<String>) Dev-Loop Late 2015 - Early 2016 @TheBigLombowski
  8. 8. Problems 1.0 1. VM images take forever to create (10 min) 2. Build fails… start over again (10 mins x Failures) 3. Developers hate dealing with Packer as part of their dev loop. 4. Shell + Terraform + Ansible stuff is pretty janky too and causes problems. 8@TheBigLombowski
  9. 9. Solutions 1.0 1. Ignore all the speed complaints! 2. Slap a fresh coat of paint on packer by hiding it in Makefiles and Gradle scripts. 9@TheBigLombowski
  10. 10. 10@TheBigLombowski
  11. 11. Reality... 11 Me CTO + Coworkers @TheBigLombowski
  12. 12. Lessons 1.0 1. I shouldn’t ignore my developers! 2. I need to listen and understand their problems better... 12@TheBigLombowski
  13. 13. Problems 1.1 and 1.2 Problem: Our devs do not always know what their problem is… Problem: I listened to our developers and we fixed the wrong problem 13@TheBigLombowski
  14. 14. 14 Dev-Test Wild West Build Build JAR (Java/Kotlin) Build VM Image Deploy Code fun main(args: Array<String>) Fix Deploy! Early 2016 @TheBigLombowski
  15. 15. 15 Dev-Test Wild West Build Build JAR (Java/Kotlin) Build VM Image Code fun main(args: Array<String>) @TheBigLombowski - Automate Workflow! - Fancy Custom Tools! Fix Deploy! Early 2016
  16. 16. ● Does all the fancy orchestration to accomplish an automated blue-green deployment. ● Solved some pain… but it also created lots of new pain and still didn’t address the real pain. ● We’re smart people though so we can fix this! 16@TheBigLombowski
  17. 17. Solution 2.0: Doubledown! 17@TheBigLombowski
  18. 18. 18 Expected Outcome... :D @TheBigLombowski
  19. 19. 19 … Actual Reaction D: @TheBigLombowski
  20. 20. Lessons 2.0 ● Thankful my bosses are patient people ;) ● We need to do a better job identifying the real pain. ● Don’t get too technology crazy. 20@TheBigLombowski
  21. 21. Solution 3.0 ● Scrap it all(!) ● Re-evaluate where things were slow ● Finally we all realized that the inability to build quickly was killing us as well. 21@TheBigLombowski
  22. 22. 22 Dev-Test Wild West Build Switch to Docker! Code func main() ... @TheBigLombowski Reboot! Early 2017 Deploy Swarm? Kubernetes? Mesos? Options!
  23. 23. 23 Dev-Test Wild West Build Switch to Docker! Code func main() ... @TheBigLombowski Reboot! Early 2017 Deploy Kubernetes!
  24. 24. Good News! ● Scrapping + Rethink a success! ● Devs Happy! Velocity increased! 24@TheBigLombowski
  25. 25. 25 Dev-Test Build WEH! This is still too slow!11!! Deploy Code Dev-Loop Mid-2017 @TheBigLombowski
  26. 26. They’re kind of Right... ● Compile code (30s+) ● Assembly binary / package (30s+) ● Build Container Image (15-30s+) ● Push Container Image (15-30s+) 26@TheBigLombowski
  27. 27. 120 seconds* @TheBigLombowski * Not scientific… possibly higher, but most likely not lower
  28. 28. ● (Java) Compile in around 30 seconds? Nope. Maven is still downloading the world 30 seconds in. ● (Python) Native dependencies require all kinds of build toolchain stuff to be installed. ● Slow networks… 28@TheBigLombowski 120 seconds is Optimistic
  29. 29. And it’s really 120s x # of Rebuilds Adds up real quick! Easy to throw away an hour or more just waiting. 29@TheBigLombowski
  30. 30. 30 Dev-Test Optimize Dockerfile Deploy Code Build Speed Attempt 1 @TheBigLombowski
  31. 31. 31 Dev-Test Multi-Stage BuildsDeploy Code Build Speed Attempt 2 @TheBigLombowski
  32. 32. 32 Dev-Test Small Base Images Deploy Code Build Speed Attempt 3 @TheBigLombowski
  33. 33. Not Enough! ● Shaves some seconds… but not enough. ● Even the fastest builds gets stuck in time sink of registry push, redeploy, registry pull. 33@TheBigLombowski
  34. 34. Faster Builds… Hail Mary Pass 1. Wrote a custom tool that did some crazy stuff around managing Docker layers and continuously rebuilding… 2. Faster builds but generated monster sized images that took ages to push and pull… so net gain of nothing. 34@TheBigLombowski
  35. 35. 35@TheBigLombowski
  36. 36. What if... Your dev machine ran your code and it just worked the same as if you ran it on Kubernetes? 36@TheBigLombowski
  37. 37. What if... 37@TheBigLombowski CNCF Sandbox! Open Source!
  38. 38. 38 Dev-Test Build + “Deploy” Restart local process Code Dev-Loop Mid-late 2017 @TheBigLombowski
  39. 39. Telepresence Topology 39@TheBigLombowski Proxy Connection Telepresence Pod replaces pod/Foo-App Foo App Bar App pod/Bar-App LB/NodePort svc/Foo-App svc/Bar-App
  40. 40. Kill Two Birds w/ One Stone ● “Deploy” is merged into Build because “Deploy” just means relaunching local process. ● Drastically simplifies the problem. 40@TheBigLombowski
  41. 41. Secondary Benefit! ● Code is on YOUR machine… so... ● You can run a debugger from your local machine easily… ● No more printf debugging and grepping logs! ○ Connect a debugger to your local process and voila breakpoints! Stack information! ○ Debugging is an often a painful part of Dev-Test so the value of this cannot be overstated. 41@TheBigLombowski
  42. 42. 42 Expected Outcome... :D @TheBigLombowski
  43. 43. 43 Actual Outcome... :D @TheBigLombowski
  44. 44. 44@TheBigLombowski
  45. 45. Not Quite... ● Devs are never satisfied! ● Build speed is great but Getting Started and Bootstrapping are real problems. 45@TheBigLombowski
  46. 46. 46 Deploy: Bootstrapping It is a big problem! Dev-Loop Late 2017 @TheBigLombowski
  47. 47. Attempt #1 ● Use Minikube on local machine ● Apply manifests of stuff ● It should all just work out right? 47@TheBigLombowski
  48. 48. Outcome... ● Minikube is fragile and tends to break in mysterious ways… ● Our deployment logic for stuff is not as clean as we would hope. Tends to also make assumptions about running from a well-configured CI env and not a dev machine. ● Developers banging their heads against their desks because of bootstrapping pain. 48@TheBigLombowski
  49. 49. Minikube is for Unicorns 49@TheBigLombowski
  50. 50. For the rest of us… remote shared environments ● If you have any cloud services you likely already have a deploy pipeline for production. ● Leverage the deploy tools to spin remote dev-test environments for individuals or teams. 50@TheBigLombowski
  51. 51. 51 Attempt #2: Kubernaut ● Remote clusters that take <5 seconds to access. ● Runs a pool of clusters in the cloud. ● Developers claim (start exclusive lock) or discard (end exclusive lock) a cluster. ● Pluggable to support a range of cluster configs and backends ● Going Open Source in 2019 @TheBigLombowski
  52. 52. 52 Dev-Test Dev-Loop Late 2017-2018 @TheBigLombowski How we (almost) do it now kubernaut claim
  53. 53. 53 Dev-Test Code Dev-Loop Late 2017-2018 @TheBigLombowski How we (almost) do it now kubernaut claim
  54. 54. 54 Dev-Test Dev-Loop Late 2017-2018 @TheBigLombowski How we (almost) do it now kubernaut claim Start telepresence Code
  55. 55. 55 Dev-Test Dev-Loop Late 2017-2018 @TheBigLombowski How we (almost) do it now kubernaut claim Start telepresence Code More Code
  56. 56. The Ugly: Dev Testing ● Dev-Testing is painful without shared remote infrastructure ● Recommended way of doing it ○ Nested API Gateways and some combination of HTTP Host/Header/Path based routing. ● Your API Gateway needs to be self-service for developers and easy to safely reconfigure. 56@TheBigLombowski
  57. 57. 57 Dev-Test Build + “Deploy” 1. Claim Cluster 2. Start Telepresence 3. Code Code Dev-Loop Late 2018 Dev Routing @TheBigLombowski
  58. 58. 58 Load Balancer Ambassador foo-serv bar-serv Ambassador foo-serv bar-serv “Prod” Kube “Dev” Kube @TheBigLombowski Our Topology
  59. 59. 59 Load Balancer Ambassador foo-serv bar-serv Ambassador foo-serv bar-serv Telepresence Proxy Prod Kube Phils Kube @TheBigLombowski Our Topology HTTP Header “Dev = Phil” Proxy Connector bar-serv Phils Laptop
  60. 60. That’s where we are ● Still plenty of work to do ● Devs are way more productive than when we started: minimal waiting for things to happen ● Keep evolving this as we scale out our team. 60@TheBigLombowski
  61. 61. Takeaways ● Rome was not built in a day ● Identify and fix the right problems! ● Identify your loops slow points… tackle low hanging fruit first. Repeat…! ● You’re most likely going to need remote development clusters for your projects. 61@TheBigLombowski
  62. 62. I will be at the booth Come and chat :) Check out Ambassador and Telepresence! Tweet me! @TheBigLombowski 62@TheBigLombowski Thanks!
  63. 63. I will be at the booth Come and chat :) Check out Ambassador and Telepresence! Tweet me! @TheBigLombowski 63@TheBigLombowski Thanks!
  64. 64. 64
  65. 65. 65