Applied Lean Startup Ideas: Continuous Deployment at kaChing
Upcoming SlideShare
Loading in...5
×
 

Applied Lean Startup Ideas: Continuous Deployment at kaChing

on

  • 11,629 views

kaChing is an online platform that connects investors with outstanding investment managers. We are a technology driven company, which has adopted lean methodologies from the start, and has achieved a ...

kaChing is an online platform that connects investors with outstanding investment managers. We are a technology driven company, which has adopted lean methodologies from the start, and has achieved a 5-minute commit-to-production cycle. Continuous deployment is a way of life and is integral to our engineering culture.

In this talk, we will present our system's architecture and discuss our service oriented platform dubbed kawala (in the process of being open sourced on http://code.google.com/p/kawala). We will describe the mechanics of an automated release from check-in to production: clean build with full regression testing in less than 3 minutes, packaging and deployment by automatically redirecting traffic using ZooKeeper for coordination, health checks and immune system to monitor the release. Finally, we will talk about planned evolutions and the next challenges we face in our infrastructure.

Statistics

Views

Total Views
11,629
Views on SlideShare
9,905
Embed Views
1,724

Actions

Likes
27
Downloads
160
Comments
1

14 Embeds 1,724

http://eng.wealthfront.com 912
http://www.slideshare.net 387
http://eng.kaching.com 188
http://www.doublecloud.org 141
https://campusvirtual.uclm.es 39
http://bibliotecaslliria.blogspot.com 28
http://bibliotecaslliria.blogspot.com.es 13
http://www.lmodules.com 6
http://feeds.feedburner.com 2
http://capellen.teletop.nl 2
http://webcache.googleusercontent.com 2
http://translate.googleusercontent.com 2
http://facebook.slideshare.com 1
https://twimg0-a.akamaihd.net 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • Wouldn't it be good if startups could get a best in class continuous deployment architecture in minutes build out of best-in-class open source hence they can focus their efforts on building innovative products. I don't know if there are many solutions available. I only know rapidklout.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Thanks to SDForum, thanks to LinkedIn for hostingToday, talk about applied lean startup and specifically about continuous deployment at kaChingDavid and I are presenting the efforts of a whole team.Tweet, share, shout, ask questionsBefore we start, let me ask you a few questions:How long from writing code to having it in production?How much time do spend on “releasing” every month?How long does deprecated code stay around?How often do you refactor code?New hire code to production?How long from product request to code in production, in front of alpha/beta testers?
  • Let me give a some context about what kaChing is.Our goal is to connect investors with outstanding investment managers.Schwab created Schawb One Source which is THE marketplace for mutual funds.We want to create the equivalent for individually managed accounts, essentially bringing wealth management to retail.On kaChing, transparency is the norm. You can know everything about the investment manager you are entrusting. His portfolio holdings, his past transaction history, his rational, philosophy. And we use all this data to objectively vet managers.In 6 months, we’ve attracted $11M and are growing very fast.We’re an SEC regulated company, en route to being FINRA regulated as a broker/dealer.
  • QA and operations engineer are typically less compensated than software engineers or architects. How can we be saving by having more expensive people do the work?Also explain what our view on QA is: two functions. Check the spec was translated to a working product correctly; Check that the product works. The first part is inherently human, the latter must be automated to eliminate waste.
  • CD’s benefits are twofold: business/lean startup and technology related.Continuous deployment is essential to driving engineering culture and enabling quick iteration cycles.Many processes which are typically seen in engineering organizing, cutting a release weekly for instance, are not needed anymore. Reduces a lot of waste, that is work not directly helping with the learning experience of creating something newWe are a financial company. We deal with people’s money. Not being able to communicate with our prime broker is a no no. Work in small batches, reduce the possibility of integration problems, and rollback as well as narrow down quickly if they ever occur.From a technology stand point, engineers are acutely aware of how things run. You are on the hook for bad code, no throwing over the wall to QA organization.Everybody needs to know about how a 24x7 site is ran.Trunk stable, we’ll talk about that in a minute
  • This is a very high level survey, the goal is to convey the necessary things to understand how our system works in the context of CD not how kaChing is built.JSON and Protobufs
  • -Scale the team: add features without adding engineers to support and maintain these features. This requires “production tests” a.k.a. monitoring.-Debugging: debugging live servers in a production environment!-At kaChing, we’ve never spent $1 on QA-Not all engineers are excited by a test-driven environment. Some are happy with “tests” taking “only” 5 hours to run.

Applied Lean Startup Ideas: Continuous Deployment at kaChing Applied Lean Startup Ideas: Continuous Deployment at kaChing Presentation Transcript

  • Applied Lean Startup Ideas:
    Continuous Deployment at kaChing
    &
    #leanstartup
  • Twenty Deployments a Day Keeps the Nasty Bugs Away
    Pascal-Louis Perez, kaChing Group Inc.
  • “Connect investors with outstandinginvestment managers”
    Pascal-Louis Perez, kaChing Group Inc.
  • Release is a Marketing ConcernReducing Code Inventory is what Engineering Focuses On
    Pascal-Louis Perez, kaChing Group Inc.
  • Lean Startup at kaChing
    “Human institution designed to create something new under conditions of extreme uncertainty” – Eric Ries
    Methodical approach to eliminating waste
    No Quality Assurance engineers, no Operations team
    Empower everybody to drive their own projects
    Pascal-Louis Perez, kaChing Group Inc.
  • What is Continuous Deployment?
    Continuous, successful and repeatable methodology to deploying code
    Automates every steps of taking checked in code and making it run on production servers, in front of customers
    Pascal-Louis Perez, kaChing Group Inc.
  • True Story
    Investment managers calls, comments about unintuitive trading flow.
    Improvements are made, and deployed within the next 20 minutes.
    We call him back “What do you think of the improvement?”
    Pascal-Louis Perez, kaChing Group Inc.
  • Benefits of Continuous Deployment
    It allows quick iterations
    Obsoletes processes, e.g. “cutting a release”
    Reduces risk
    Everyone is aware of production
    No one throws code over the wall
    Exposes 24x7 operational requirements
    Trunk stable
    Pascal-Louis Perez, kaChing Group Inc.
  • ContinuousDeployment
    Immune System
    Making Continuous Deployment a Reality
    Pascal-Louis Perez, kaChing Group Inc.
    Culture
    Test Driven Development
  • #architecture
    Pascal-Louis Perez, kaChing Group Inc.
  • Our architecture
    Service oriented system
    Vertical sharding
    Everything uses the same platform kawala
    Coordination using ZooKeeper
    Data interchange using JSON and Protobufs
    Pascal-Louis Perez, kaChing Group Inc.
  • Typical Stack
    Clustered services, multiple instances
    Replicated databases (e.g. MySQL or NoSQL)
    Caching (e.g. memcached)
    Denormalized data (e.g. Voldemort)
    Pascal-Louis Perez, kaChing Group Inc.
  • kawala
    Command pattern. A command is a query
    Dynamic Inversion of Control
    Queries produce a value
    Pascal-Louis Perez, kaChing Group Inc.
  • Sample Query
    Pascal-Louis Perez, kaChing Group Inc.
    class GetUser(Id<User> userId) { @Inject UserRepository repository; public User process() {
    return repository.load(userId); }}
  • What We do with Queries
    Pascal-Louis Perez, kaChing Group Inc.
    Persist Queries
    RPC
    Command Line Tools
    Serve BLOBS
    CPS
    Render Pages
    Push into MQ
  • Lifecycle
    1. Commit
    2. Testing
    3. Deployment
    4. Production Validation
    5. Lifetime Monitoring
  • Development
    Trunk stable
    Small, frequent commits
    Unreleased features live behind experiments
    Forward & Backward compatibility
    Responsive code review
    Pair difficult problems
    Trivial rollbacks
  • #testing
  • Testing Philosophy
    Only automated testing matters
    If it isn’t tested, it isn’t finished or correct
    Write testable code
    Embrace abstractions
    Testing is cross functional, we all own quality
  • Benefits of TDD
    It allows quick iterations
    It empowers engineers to change anything, and as such helps in scaling the team
    It is more cost effective than debugging
    It obsoletes the need for functional QA
    It facilitates continuous refactoring, allowing the code to get better with age
    It attracts the right kind of engineers
    Pascal-Louis Perez, kaChing Group Inc.
  • Types of Testing
    Unit Testing - does the code work?
    Integration Testing - does the code work together?
    Regression Testing - learn from your mistakes
    Frontend Testing - a whole different ballgame
  • Defensive Testing
    Nightmare scenarios
    A common conversation at lunch:
    Alice: What would happen if X blew up?
    Bob: uh... the site would go down.
    Fix it, test for it
  • Implementation
    Frameworks: junit, dbunit
    CI: Hudson
    Frontend: Selenium
  • Production
    Process, Infrastructure, Deployment, Monitoring
  • Process
    Everyone: writes code and tests, releases to production, adds monitoring
    Specialists help out when needed
    New hires push code the first day
    Problems, issues, errors, bugs, oversights...
  • Stability
    Cluster everything
    As little global state as possible
    Maintain global state through ZooKeeper
    Monitor everything
  • ZooKeeper
    Reliable Distributed Synchronization
    For:
    Service Discovery
    Service Status
    Coarse locking
  • Life of a Deployment
    Pascal-Louis Perez, kaChing Group Inc.
    DM
    PM
    PM
    PM
    ZK
  • Life of a Deployment
    Pascal-Louis Perez, kaChing Group Inc.
    DM
    PM
    PM
    PM
    ZK
  • Life of a Deployment
    Pascal-Louis Perez, kaChing Group Inc.
    DM
    PM
    PM
    ZK
  • Life of a Deployment
    Pascal-Louis Perez, kaChing Group Inc.
    DM
    PM
    PM
    PM
    ZK
  • Life of a Deployment
    Pascal-Louis Perez, kaChing Group Inc.
    DM
    PM
    PM
    PM
    ZK
  • Deployment
    Exponentially increasing group sizes
    Increased monitoring during deployment
    Self Test
    Automated rollbacks
  • Implementation
    Building: ant
    Deploying: rpm, yum
    Monitoring: nagios, jcollectd, RRDtool, jmx
    Custom automation
  • #monitoring
  • Monitoring Philosophy
    Prefer business metrics
    Monitor statistical deviations not absolute values
    Automatically annotate graphs
  • Monitoring Errors
    False negatives - errors of omission
    False positives - errors of implementation
  • End to End Monitoring
    Run Selenium on production
    Accessibility, Speed
    Ad-hoc Keynote - customized for our flows
    Must control data creation, analytics impact
  • #future-developments
    Pascal-Louis Perez, kaChing Group Inc.
  • Quarantining
    Isolate new releases
    Flexible partitioning of requests
    Gradually shift load to fresh services
    Pascal-Louis Perez, kaChing Group Inc.
  • Describing Infrastructure
    Many moving parts
    nagios, collectd, backups, services, databases, …
    Consistency is key
    Adding new tools should be easy and thorough
    Standardize best practices
    Pascal-Louis Perez, kaChing Group Inc.
  • What we Covered Today
    Lean Startup and Continuous Deployment
    Anatomy of a Release
    Commit, Testing, Deployment, Monitoring
    Future Developments
    Quarantining, DSL for Infrastructure
    Pascal-Louis Perez, kaChing Group Inc.
  • References
    We’re Recruiting jobs@kaching.com
    kaChing’s blog http://eng.kaching.com
    kawalahttp://bit.ly/kawala
    Deployment Infrastructure http://eng.kaching.com/2010/05/deployment-infrastructure-for.html
    Extreme Testing http://bit.ly/9bOFaA
    Writing Testable Codehttp://googletesting.blogspot.com/2008/08/by-miko-hevery-so-you-decided-to.html
    ZooKeeperhttp://bit.ly/kc-zookeeper
    Eric Rieshttp://startuplessonslearned.com
    Pascal-Louis Perez, kaChing Group Inc.