INNOVATING FASTER
Supporting microservices w/ SBT, Continuous Delivery & LXC
Kevin Scaldeferri
Gilt Groupe
June 18, 2013

CULTURE
• Rapid feature development
• Rapid experimentation
• Find what works, discard what doesn’t

INTHE BEGINNING

ONE (REALLY) BIG RAILS APP
• ~1000 models and controllers
• ~ 200k lines of Ruby
• ~ 50k lines of PostgreSQL stored procedures

PROS OF MONOLITHIC
DESIGN
• Rapid (Initial) Development
• Simple development
• Simple deployment
• Simple operational model

CONS OF MONOLITHIC
DESIGN
• Unclear Ownership
• Complex Dependencies
• LengthyTest Cycles
• Unexpected Performance Impacts

DAY OF RECKONING

THE LOUBOUTIN INCIDENT

GETTING
(THE MICROSERVICES)
RELIGION

EARLY DAYS
• Pure Java
• Moving fast, lots of C&P
• Lots of snowflakes and competing approaches
• Ant-based build installed via git submodule
• Capistrano for deployment

DIGRESSION ON GIT
SUBMODULES

DOWNSIDES OF ANT
• Build scripts in a different language
• Hard to integrate many tools
• Impedance mismatch w/ Scala

SBT

WHAT’S COOL ABOUT SBT
• Fast incremental builds of Scala & Java
• ~compile, ~test
• Interactive shell & console
• Configuration is “just” Scala code

BUT...
• Very sophisticated use of Scala
• Subtle mental model
• Can be hard to debug
• Lots of historical baggage in our build & deployment processes

GILT-SBT-BUILD
• One big SBT plugin pulling in all other plugins and
configuration
• Plugin is versioned just like any other software
• (Actually builds itself!)
• Super simple config for individual services

gilt.GiltProject.jarSettings
name
:=
"lib-­‐jenkins"
libraryDependencies
++=
Seq(
"net.databinder"
%%
"dispatch-­‐core"
%
"0.8.8",
"net.databinder"
%%
"dispatch-­‐http"
%
"0.8.8"
)

PLUGIN PROVIDES
• Nexus config
• Testing & Coverage Libraries
• RPMs
• Standard Run Scripts
• NewRelic Support
• Release Emails
• SemVer Analysis
• Add’l Dependency
Heuristics
• Integration Builds
• ... and more....

WHAT ABOUT MULTI-
PROJECT BUILDS?

object
Build
extends
ClientServerCoreProject
{
val
name
=
"svc-­‐persistent-­‐session"
val
coreDeps
=
...
val
serverDeps
=
...
val
clientDeps
=
...
override
val
ioncannonTrack
=
IonCannon.FastTrack
}

IONCANNON?

CONTINUOUS DELIVERY
• At the granularity of released versions
• Too many commits and too many services to deploy every
commit

DEVELOPMENT
• Teams have a shared dev / test environment
• Develop locally w/ tunnels to services
• Submit to Gerrit for peer review
• Deploy & test on team environment
• ‘sbt release’ when ready

FAST-TRACK
• Fully automated testing & deployment
• ‘sbt release’ triggers deployment to a staging environment
mirroring production
• Automated tests run
• Promote to production or rollback

SKYPE NOTIFICATIONS

FAST-TRACK
Launched in November 2012. Currently over
100 services and applications.

SLOW-TRACK
• Automated Deployment
• ManualTesting / Approval

SLOW-TRACK IN PRACTICE
• Dramatically less adoption than fast-track
• Systems without good automated tests tend to be older
systems which also haven’t upgraded to the current build &
deploy toolchain
• Teams do manual testing on test environments already
• Better tools for UI testing tend to eliminate the need for this

SELENIUM:TEST
• ‘sbt selenium:test’
• Built on ScalaTest 2.0 Selenium DSL
• Automated browser testing with reusable components
• All tests written in Scala
• Tests can be run in any environment
• Selenium grid available

@Selenium
class
Example
extends
FlatSpecTestBase
with
Matchers
with
ConfigurableBrowser
with
LoggedTestUser
with
OnProductDetailPage
with
AvailableToPurchaseItem
with
InMens
with
CloseBrowserAfterAllTests
{
"A
size
chart"
should
"be
available"
in
{
//
test
goes
here
}
}

DEPLOYMENT

AN APOLOGY

CURRENT DEPLOYMENT
• Bare metal machines
• Multiple services per machine
• Highly manual process to balance resource requirements

THE FUTURE
• Deploy to immutable LXC containers, one service per
container
• New versions deploy to fresh containers, old containers are
retired
• Services specify resource requirements, system finds available
capacity

CLOUDSTACK
• Private cloud management
• Also evaluated OpenStack & Ganeti
• CloudStack’s APIs seemed best suited to our design
• Added LXC support to CloudStack 4.2

WHY LXC?

DEPLOYMENT SEQUENCE
• svc-software-provisioning - create new container(s)
• install new version onto containers
• svc-loadbalancer - move traffic to new version
• svc-software-provisioning - delete old containers

SUMMARY
• Support rapid development of micro-services and micro-
applications via:
• Powerful & highly abstracted build system
• Continuous delivery & great tools for automated testing
• (Upcoming) Simple & consistent hardware provisioning

THANKYOU

QUESTIONS?