The document discusses the transition from a monolithic Rails application to a microservices architecture using sbt and LXC for continuous delivery. It highlights the challenges and benefits of both approaches, including the deployment process and automated testing strategies. The author emphasizes the need for a powerful build system and efficient resource management to support rapid development and deployment of services.

1. INNOVATING FASTER
Supporting microservices w/ SBT, Continuous Delivery & LXC
Kevin Scaldeferri
Gilt Groupe
June 18, 2013

4. CULTURE
• Rapid feature development
• Rapid experimentation
• Find what works, discard what doesn’t

5. INTHE BEGINNING

6. ONE (REALLY) BIG RAILS APP
• ~1000 models and controllers
• ~ 200k lines of Ruby
• ~ 50k lines of PostgreSQL stored procedures

7. PROS OF MONOLITHIC
DESIGN
• Rapid (Initial) Development
• Simple development
• Simple deployment
• Simple operational model

8. CONS OF MONOLITHIC
DESIGN
• Unclear Ownership
• Complex Dependencies
• LengthyTest Cycles
• Unexpected Performance Impacts

9. DAY OF RECKONING

10. THE LOUBOUTIN INCIDENT

11. GETTING
(THE MICROSERVICES)
RELIGION

12. EARLY DAYS
• Pure Java
• Moving fast, lots of C&P
• Lots of snowflakes and competing approaches
• Ant-based build installed via git submodule
• Capistrano for deployment

13. DIGRESSION ON GIT
SUBMODULES

14. DOWNSIDES OF ANT
• Build scripts in a different language
• Hard to integrate many tools
• Impedance mismatch w/ Scala

15. SBT

16. WHAT’S COOL ABOUT SBT
• Fast incremental builds of Scala & Java
• ~compile, ~test
• Interactive shell & console
• Configuration is “just” Scala code

17. BUT...
• Very sophisticated use of Scala
• Subtle mental model
• Can be hard to debug
• Lots of historical baggage in our build & deployment processes

18. GILT-SBT-BUILD
• One big SBT plugin pulling in all other plugins and
configuration
• Plugin is versioned just like any other software
• (Actually builds itself!)
• Super simple config for individual services

19. gilt.GiltProject.jarSettings
name
:=
"lib-­‐jenkins"
libraryDependencies
++=
Seq(
"net.databinder"
%%
"dispatch-­‐core"
%
"0.8.8",
"net.databinder"
%%
"dispatch-­‐http"
%
"0.8.8"
)

20. PLUGIN PROVIDES
• Nexus config
• Testing & Coverage Libraries
• RPMs
• Standard Run Scripts
• NewRelic Support
• Release Emails
• SemVer Analysis
• Add’l Dependency
Heuristics
• Integration Builds
• ... and more....

21. WHAT ABOUT MULTI-
PROJECT BUILDS?

22. object
Build
extends
ClientServerCoreProject
{
val
name
=
"svc-­‐persistent-­‐session"
val
coreDeps
=
...
val
serverDeps
=
...
val
clientDeps
=
...
override
val
ioncannonTrack
=
IonCannon.FastTrack
}

23. IONCANNON?

24. CONTINUOUS DELIVERY
• At the granularity of released versions
• Too many commits and too many services to deploy every
commit

25. DEVELOPMENT
• Teams have a shared dev / test environment
• Develop locally w/ tunnels to services
• Submit to Gerrit for peer review
• Deploy & test on team environment
• ‘sbt release’ when ready

26. FAST-TRACK
• Fully automated testing & deployment
• ‘sbt release’ triggers deployment to a staging environment
mirroring production
• Automated tests run
• Promote to production or rollback

28. SKYPE NOTIFICATIONS

29. FAST-TRACK
Launched in November 2012. Currently over
100 services and applications.

30. SLOW-TRACK
• Automated Deployment
• ManualTesting / Approval

32. SLOW-TRACK IN PRACTICE
• Dramatically less adoption than fast-track
• Systems without good automated tests tend to be older
systems which also haven’t upgraded to the current build &
deploy toolchain
• Teams do manual testing on test environments already
• Better tools for UI testing tend to eliminate the need for this

33. SELENIUM:TEST
• ‘sbt selenium:test’
• Built on ScalaTest 2.0 Selenium DSL
• Automated browser testing with reusable components
• All tests written in Scala
• Tests can be run in any environment
• Selenium grid available

34. @Selenium
class
Example
extends
FlatSpecTestBase
with
Matchers
with
ConfigurableBrowser
with
LoggedTestUser
with
OnProductDetailPage
with
AvailableToPurchaseItem
with
InMens
with
CloseBrowserAfterAllTests
{
"A
size
chart"
should
"be
available"
in
{
//
test
goes
here
}
}

35. DEPLOYMENT

36. AN APOLOGY

37. CURRENT DEPLOYMENT
• Bare metal machines
• Multiple services per machine
• Highly manual process to balance resource requirements

38. THE FUTURE
• Deploy to immutable LXC containers, one service per
container
• New versions deploy to fresh containers, old containers are
retired
• Services specify resource requirements, system finds available
capacity

39. CLOUDSTACK
• Private cloud management
• Also evaluated OpenStack & Ganeti
• CloudStack’s APIs seemed best suited to our design
• Added LXC support to CloudStack 4.2

40. WHY LXC?

41. DEPLOYMENT SEQUENCE
• svc-software-provisioning - create new container(s)
• install new version onto containers
• svc-loadbalancer - move traffic to new version
• svc-software-provisioning - delete old containers

42. SUMMARY
• Support rapid development of micro-services and micro-
applications via:
• Powerful & highly abstracted build system
• Continuous delivery & great tools for automated testing
• (Upcoming) Simple & consistent hardware provisioning

43. THANKYOU

44. QUESTIONS?