Innovating faster with SBT, Continuous Delivery, and LXC

8,122 views

Published on

A case study of the tools and techniques used at Gilt Groupe to develop and deploy a system composed of over 200 micro-services.

Published in: Technology

Innovating faster with SBT, Continuous Delivery, and LXC

  1. 1. INNOVATING FASTERSupporting microservices w/ SBT, Continuous Delivery & LXCKevin ScaldeferriGilt GroupeJune 18, 2013
  2. 2. CULTURE• Rapid feature development• Rapid experimentation• Find what works, discard what doesn’t
  3. 3. INTHE BEGINNING
  4. 4. ONE (REALLY) BIG RAILS APP• ~1000 models and controllers• ~ 200k lines of Ruby• ~ 50k lines of PostgreSQL stored procedures
  5. 5. PROS OF MONOLITHICDESIGN• Rapid (Initial) Development• Simple development• Simple deployment• Simple operational model
  6. 6. CONS OF MONOLITHICDESIGN• Unclear Ownership• Complex Dependencies• LengthyTest Cycles• Unexpected Performance Impacts
  7. 7. DAY OF RECKONING
  8. 8. THE LOUBOUTIN INCIDENT
  9. 9. GETTING(THE MICROSERVICES)RELIGION
  10. 10. EARLY DAYS• Pure Java• Moving fast, lots of C&P• Lots of snowflakes and competing approaches• Ant-based build installed via git submodule• Capistrano for deployment
  11. 11. DIGRESSION ON GITSUBMODULES
  12. 12. DOWNSIDES OF ANT• Build scripts in a different language• Hard to integrate many tools• Impedance mismatch w/ Scala
  13. 13. SBT
  14. 14. WHAT’S COOL ABOUT SBT• Fast incremental builds of Scala & Java• ~compile, ~test• Interactive shell & console• Configuration is “just” Scala code
  15. 15. BUT...• Very sophisticated use of Scala• Subtle mental model• Can be hard to debug• Lots of historical baggage in our build & deployment processes
  16. 16. GILT-SBT-BUILD• One big SBT plugin pulling in all other plugins andconfiguration• Plugin is versioned just like any other software• (Actually builds itself!)• Super simple config for individual services
  17. 17. gilt.GiltProject.jarSettingsname  :=  "lib-­‐jenkins"libraryDependencies  ++=  Seq(    "net.databinder"  %%  "dispatch-­‐core"  %  "0.8.8",    "net.databinder"  %%  "dispatch-­‐http"  %  "0.8.8")
  18. 18. PLUGIN PROVIDES• Nexus config• Testing & Coverage Libraries• RPMs• Standard Run Scripts• NewRelic Support• Release Emails• SemVer Analysis• Add’l DependencyHeuristics• Integration Builds• ... and more....
  19. 19. WHAT ABOUT MULTI-PROJECT BUILDS?
  20. 20. object  Build  extends  ClientServerCoreProject  {    val  name  =  "svc-­‐persistent-­‐session"    val  coreDeps  =  ...    val  serverDeps  =  ...    val  clientDeps  =  ...    override  val  ioncannonTrack  =  IonCannon.FastTrack}
  21. 21. IONCANNON?
  22. 22. CONTINUOUS DELIVERY• At the granularity of released versions• Too many commits and too many services to deploy everycommit
  23. 23. DEVELOPMENT• Teams have a shared dev / test environment• Develop locally w/ tunnels to services• Submit to Gerrit for peer review• Deploy & test on team environment• ‘sbt release’ when ready
  24. 24. FAST-TRACK• Fully automated testing & deployment• ‘sbt release’ triggers deployment to a staging environmentmirroring production• Automated tests run• Promote to production or rollback
  25. 25. SKYPE NOTIFICATIONS
  26. 26. FAST-TRACKLaunched in November 2012. Currently over100 services and applications.
  27. 27. SLOW-TRACK• Automated Deployment• ManualTesting / Approval
  28. 28. SLOW-TRACK IN PRACTICE• Dramatically less adoption than fast-track• Systems without good automated tests tend to be oldersystems which also haven’t upgraded to the current build &deploy toolchain• Teams do manual testing on test environments already• Better tools for UI testing tend to eliminate the need for this
  29. 29. SELENIUM:TEST• ‘sbt selenium:test’• Built on ScalaTest 2.0 Selenium DSL• Automated browser testing with reusable components• All tests written in Scala• Tests can be run in any environment• Selenium grid available
  30. 30. @Seleniumclass  Example  extends  FlatSpecTestBasewith  Matchers  with  ConfigurableBrowser  with  LoggedTestUserwith  OnProductDetailPagewith  AvailableToPurchaseItemwith  InMenswith  CloseBrowserAfterAllTests  {    "A  size  chart"  should  "be  available"  in  {      //  test  goes  here    }}
  31. 31. DEPLOYMENT
  32. 32. AN APOLOGY
  33. 33. CURRENT DEPLOYMENT• Bare metal machines• Multiple services per machine• Highly manual process to balance resource requirements
  34. 34. THE FUTURE• Deploy to immutable LXC containers, one service percontainer• New versions deploy to fresh containers, old containers areretired• Services specify resource requirements, system finds availablecapacity
  35. 35. CLOUDSTACK• Private cloud management• Also evaluated OpenStack & Ganeti• CloudStack’s APIs seemed best suited to our design• Added LXC support to CloudStack 4.2
  36. 36. WHY LXC?
  37. 37. DEPLOYMENT SEQUENCE• svc-software-provisioning - create new container(s)• install new version onto containers• svc-loadbalancer - move traffic to new version• svc-software-provisioning - delete old containers
  38. 38. SUMMARY• Support rapid development of micro-services and micro-applications via:• Powerful & highly abstracted build system• Continuous delivery & great tools for automated testing• (Upcoming) Simple & consistent hardware provisioning
  39. 39. THANKYOU
  40. 40. QUESTIONS?

×