Iterate Like a Whirling Dervish

Continuous Deployment at kaChing
#leanstartup

“Connect investors with outstandinginvestment managers”
Pascal-Louis Perez, kaChing Group Inc.

Assets Managed by Mutual Funds in the US
$11,000,000,000,000

Pivot
Our vision of bringing the quality of Ivy League Endowment management to retail is unaltered
Powering a marketplace where customers entrust high end managers with their money in separately managed accounts. We are SEC & FINRA regulated.
Three years ago, kaChing was a fantasy stock trading app for young adults on facebook. Our strategy was to create an eco system where avid amateurs could rise through the ranks and make it to pro league.

Constant tension between product and development.
Right architecture only in the context of how it is used. Initially, you don’t know for sure how something will be used!
We’ve adopted proportional investing
Build cheap
When you spend X hours of time fixing problems
Spend a corresponding X hours improving the feature
Build “Right”
Features

Strive for 80% of the value for 20% of the cost
Realize that 80% of what you build will go to waste. Validate before investing more time!
80/20

Errors Are Costly!
Continuous Deployment is a mission critical way of running our business which provides tremendous value
Errors are costly
Brand impact
SLA penalties
And when you’re regulated, there’s also liabilities
Turning the wheels on the software with high confidence provides tremendous business value

Deploy Fast and Often

What is Continuous Deployment?
Continuous, successful and repeatable methodology to deploying code
Automates every steps of taking checked in code and making it run on production servers, used by customers

e.g. Twenty Deployments a Day

Release is a Marketing ConcernReducing Code Inventory is Engineering’s Job

Homage to Gursky: 'Kirksky‘copyright 2010 by Kirk Crippens
Supermarket Analogy
(Think Safeway)
You don’t close the store to fill the shelves!
Continuously get fresh merchandise out
For product releases, simply close or hide the aisle

Traditional Release Organization
Software organized as a tree
Trunk is what goes on production, it is The Truth
Engineers work on features in branches
Features get integrated into trunk
Release train on regular cyclic period
QA ensures correctness of each train
P1 bugs get patched back into the train
Flip the switch release

Timeline
QA
Stage
Release
Fix P1 Bugs
Development
Cut a Release
Integrate Patches
1-4 Weeks

Timeline, Revisited
Deployment
Automated
QA
Stage
Fix P1 Bugs
Release
Development
or Testing
Cut a Release
Integrate Patches
Monitoring
Experiments
5-10 Minutes

Experiments
Split features which are not finished“if press person, show the new home page”if (isInExperiment(NEW_HOME_PAGE)) { …} else { …}
(Yes, this is just like A/B Testing)

True Story
Investment managers calls, comments about unintuitive trading flow.
Improvements are made, and deployed within the next 20 minutes.
We call him back “What do you think of the improvement?”

Benefits of Continuous Deployment
It allows quick iterations
Obsoletes processes, e.g. “cutting a release”
Reduces risk
Everyone is aware of production
No one throws code over the wall
Exposes 24x7 operational requirements
Trunk stable

Continuous Deployment
ImmuneSystem
Culture
ContinuousIntegration
Test DrivenDevelopment

kaChing’s Architecture
Service oriented system
Vertical sharding
Everything uses the same platform kawala
Coordination using ZooKeeper
Data interchange using JSON and Protobufs

Typical Stack
Clustered services, multiple instances
Replicated databases (e.g. MySQL, Redis, …)
Caching (e.g. memcached)
Denormalized data (e.g. Voldemort)

Timeline, Revisited
Development
Deployment
Testing
Monitoring

Development
Trunk stable
Small, frequent commits
Unreleased features live behind experiments
Forward & Backward compatibility
Responsive code review
Pair difficult problems
Trivial rollbacks

Testing Philosophy
Only automated testing matters
If it isn’t tested, it isn’t finished or correct
Write testable code
Embrace abstractions
Testing is cross functional, we all own quality

Benefits of TDD
It allows quick iterations
It empowers engineers to change anything, and as such helps in scaling the team
It is more cost effective than debugging
It obsoletes the need for functional QA
It facilitates continuous refactoring, allowing the code to get better with age
It attracts the right kind of engineers

Types of Testing
Unit Testing - does the code work?
Integration Testing - does the code work together?
Regression Testing - learn from your mistakes
Frontend Testing - a whole different ballgame

Defensive Testing
Capture invariants in tests
Nightmare scenarios
A common conversation at lunch:
Alice: What would happen if X blew up?
Bob: uh... the site would go down.
Fix it, test for it

Implementation
Frameworks: junit, dbunit
CI: Hudson
Frontend: Selenium

Life of a Deployment
DM
PM
PM
PM
ZK

DM
PM
PM
ZK

DM
PM
PM
PM
ZK

Deployment
Exponentially increasing group sizes
Increased monitoring during deployment
Self Test
Automated rollbacks

Stability
Cluster everything
As little global state as possible
Maintain global state through ZooKeeper
Monitor everything

ZooKeeper
Reliable Distributed Synchronization
Service Discovery
Service Status
Coarse locking

Implementation
Building: ant
Deploying: rpm, yum
Monitoring: nagios, jcollectd, RRDtool, jmx
Custom automation

Monitoring Philosophy
Prefer business metrics
Monitor statistical deviations not absolute values
Automatically annotate graphs

Monitoring Errors
False negatives - errors of omission
False positives - errors of implementation

End to End Monitoring
Run Selenium on production
Accessibility, Speed
Ad-hoc Keynote - customized for our flows
Must control data creation, analytics impact

#future-developments

Quarantining
Isolate new releases
Flexible partitioning of requests
Gradually shift load to fresh services

Describing Infrastructure
Many moving parts
nagios, collectd, backups, services, databases, …
Consistency is key
Adding new tools should be easy and thorough
Standardize best practices

TGIF?
We will be hosting a TGIF in August in our office@pascallouis / @kachingeng

References
We’re Recruiting jobs@kaching.com
kaChing’s blog http://eng.kaching.com
kawalahttp://bit.ly/kawala
Deployment Infrastructure http://eng.kaching.com/2010/05/deployment-infrastructure-for.html
Extreme Testing http://bit.ly/9bOFaA
Writing Testable Codehttp://googletesting.blogspot.com/2008/08/by-miko-hevery-so-you-decided-to.html
The Testability Explorer Bloghttp://misko.hevery.com
ZooKeeperhttp://bit.ly/kc-zookeeper
Lessons Learned http://startuplessonslearned.com

http://eng.wealthfront.com	615
http://torgronsund.wordpress.com	81
http://eng.kaching.com	79
http://torgronsund.com	48
http://www.encob.net	12
http://www.techgig.com	11
http://www.investmentor.com	4
http://www.foundergems.com	2
http://www.slideshare.net	2
http://leanstartups.wpengine.com	1
http://www.onlydoo.com	1
http://www.leanmovement.com	1
http://www.hooeeywebprint.com	1
http://www.techgig.timesjobs.com	1
http://www.hanrss.com	1

Iterate Like a Whirling Dervish

by Pascal-Louis Perez

Accessibility

Categories

Upload Details

Usage Rights

15 Embeds 860

Statistics