Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Applied Lean Startup Ideas: Continuous Deployment at kaChing


Published on

kaChing is an online platform that connects investors with outstanding investment managers. We are a technology driven company, which has adopted lean methodologies from the start, and has achieved a 5-minute commit-to-production cycle. Continuous deployment is a way of life and is integral to our engineering culture.

In this talk, we will present our system's architecture and discuss our service oriented platform dubbed kawala (in the process of being open sourced on We will describe the mechanics of an automated release from check-in to production: clean build with full regression testing in less than 3 minutes, packaging and deployment by automatically redirecting traffic using ZooKeeper for coordination, health checks and immune system to monitor the release. Finally, we will talk about planned evolutions and the next challenges we face in our infrastructure.

Published in: Technology
  • Wouldn't it be good if startups could get a best in class continuous deployment architecture in minutes build out of best-in-class open source hence they can focus their efforts on building innovative products. I don't know if there are many solutions available. I only know rapidklout.
    Are you sure you want to  Yes  No
    Your message goes here

Applied Lean Startup Ideas: Continuous Deployment at kaChing

  1. Applied Lean Startup Ideas:<br />Continuous Deployment at kaChing<br />&<br />#leanstartup<br />
  2. Twenty Deployments a Day Keeps the Nasty Bugs Away<br />Pascal-Louis Perez, kaChing Group Inc.<br />
  3. “Connect investors with outstandinginvestment managers”<br />Pascal-Louis Perez, kaChing Group Inc.<br />
  4. Release is a Marketing ConcernReducing Code Inventory is what Engineering Focuses On<br />Pascal-Louis Perez, kaChing Group Inc.<br />
  5. Lean Startup at kaChing<br />“Human institution designed to create something new under conditions of extreme uncertainty” – Eric Ries<br />Methodical approach to eliminating waste<br />No Quality Assurance engineers, no Operations team<br />Empower everybody to drive their own projects<br />Pascal-Louis Perez, kaChing Group Inc.<br />
  6. What is Continuous Deployment?<br />Continuous, successful and repeatable methodology to deploying code<br />Automates every steps of taking checked in code and making it run on production servers, in front of customers<br />Pascal-Louis Perez, kaChing Group Inc.<br />
  7. True Story<br />Investment managers calls, comments about unintuitive trading flow.<br />Improvements are made, and deployed within the next 20 minutes.<br />We call him back “What do you think of the improvement?”<br />Pascal-Louis Perez, kaChing Group Inc.<br />
  8. Benefits of Continuous Deployment<br />It allows quick iterations<br />Obsoletes processes, e.g. “cutting a release”<br />Reduces risk<br />Everyone is aware of production<br />No one throws code over the wall<br />Exposes 24x7 operational requirements<br />Trunk stable<br />Pascal-Louis Perez, kaChing Group Inc.<br />
  9. ContinuousDeployment<br />Immune System<br />Making Continuous Deployment a Reality<br />Pascal-Louis Perez, kaChing Group Inc.<br />Culture<br />Test Driven Development<br />
  10. #architecture<br />Pascal-Louis Perez, kaChing Group Inc.<br />
  11. Our architecture<br />Service oriented system<br />Vertical sharding<br />Everything uses the same platform kawala<br />Coordination using ZooKeeper<br />Data interchange using JSON and Protobufs<br />Pascal-Louis Perez, kaChing Group Inc.<br />
  12. Typical Stack<br />Clustered services, multiple instances<br />Replicated databases (e.g. MySQL or NoSQL)<br />Caching (e.g. memcached)<br />Denormalized data (e.g. Voldemort)<br />Pascal-Louis Perez, kaChing Group Inc.<br />
  13. kawala<br />Command pattern. A command is a query<br />Dynamic Inversion of Control<br />Queries produce a value<br />Pascal-Louis Perez, kaChing Group Inc.<br />
  14. Sample Query<br />Pascal-Louis Perez, kaChing Group Inc.<br />class GetUser(Id<User> userId) { @Inject UserRepository repository; public User process() {<br /> return repository.load(userId); }}<br />
  15. What We do with Queries<br />Pascal-Louis Perez, kaChing Group Inc.<br />Persist Queries<br />RPC<br />Command Line Tools<br />Serve BLOBS<br />CPS<br />Render Pages<br />Push into MQ<br />
  16. Lifecycle<br />1. Commit<br />2. Testing<br />3. Deployment<br />4. Production Validation<br />5. Lifetime Monitoring<br />
  17. Development<br />Trunk stable<br />Small, frequent commits<br />Unreleased features live behind experiments<br />Forward & Backward compatibility<br />Responsive code review<br />Pair difficult problems<br />Trivial rollbacks<br />
  18. #testing<br />
  19. Testing Philosophy<br />Only automated testing matters<br />If it isn’t tested, it isn’t finished or correct<br />Write testable code<br />Embrace abstractions<br />Testing is cross functional, we all own quality<br />
  20. Benefits of TDD<br />It allows quick iterations<br />It empowers engineers to change anything, and as such helps in scaling the team<br />It is more cost effective than debugging<br />It obsoletes the need for functional QA<br />It facilitates continuous refactoring, allowing the code to get better with age<br />It attracts the right kind of engineers<br />Pascal-Louis Perez, kaChing Group Inc.<br />
  21. Types of Testing<br />Unit Testing - does the code work?<br />Integration Testing - does the code work together?<br />Regression Testing - learn from your mistakes<br />Frontend Testing - a whole different ballgame<br />
  22. Defensive Testing<br />Nightmare scenarios<br />A common conversation at lunch:<br />Alice: What would happen if X blew up?<br />Bob: uh... the site would go down.<br />Fix it, test for it<br />
  23. Implementation<br />Frameworks: junit, dbunit<br />CI: Hudson<br />Frontend: Selenium<br />
  24. Production<br />Process, Infrastructure, Deployment, Monitoring<br />
  25. Process<br />Everyone: writes code and tests, releases to production, adds monitoring<br />Specialists help out when needed<br />New hires push code the first day<br />Problems, issues, errors, bugs, oversights...<br />
  26. Stability<br />Cluster everything<br />As little global state as possible<br />Maintain global state through ZooKeeper<br />Monitor everything<br />
  27. ZooKeeper<br />Reliable Distributed Synchronization<br />For:<br />Service Discovery<br />Service Status<br />Coarse locking<br />
  28. Life of a Deployment<br />Pascal-Louis Perez, kaChing Group Inc.<br />DM<br />PM<br />PM<br />PM<br />ZK<br />
  29. Life of a Deployment<br />Pascal-Louis Perez, kaChing Group Inc.<br />DM<br />PM<br />PM<br />PM<br />ZK<br />
  30. Life of a Deployment<br />Pascal-Louis Perez, kaChing Group Inc.<br />DM<br />PM<br />PM<br />ZK<br />
  31. Life of a Deployment<br />Pascal-Louis Perez, kaChing Group Inc.<br />DM<br />PM<br />PM<br />PM<br />ZK<br />
  32. Life of a Deployment<br />Pascal-Louis Perez, kaChing Group Inc.<br />DM<br />PM<br />PM<br />PM<br />ZK<br />
  33. Deployment<br />Exponentially increasing group sizes<br />Increased monitoring during deployment<br />Self Test<br />Automated rollbacks<br />
  34. Implementation<br />Building: ant<br />Deploying: rpm, yum<br />Monitoring: nagios, jcollectd, RRDtool, jmx<br />Custom automation<br />
  35. #monitoring<br />
  36. Monitoring Philosophy<br />Prefer business metrics<br />Monitor statistical deviations not absolute values<br />Automatically annotate graphs<br />
  37. Monitoring Errors<br />False negatives - errors of omission<br />False positives - errors of implementation<br />
  38. End to End Monitoring<br />Run Selenium on production<br />Accessibility, Speed<br />Ad-hoc Keynote - customized for our flows<br />Must control data creation, analytics impact<br />
  39. #future-developments<br />Pascal-Louis Perez, kaChing Group Inc.<br />
  40. Quarantining<br />Isolate new releases<br />Flexible partitioning of requests<br />Gradually shift load to fresh services<br />Pascal-Louis Perez, kaChing Group Inc.<br />
  41. Describing Infrastructure<br />Many moving parts<br />nagios, collectd, backups, services, databases, …<br />Consistency is key<br />Adding new tools should be easy and thorough<br />Standardize best practices<br />Pascal-Louis Perez, kaChing Group Inc.<br />
  42. What we Covered Today<br />Lean Startup and Continuous Deployment<br />Anatomy of a Release<br />Commit, Testing, Deployment, Monitoring<br />Future Developments<br />Quarantining, DSL for Infrastructure<br />Pascal-Louis Perez, kaChing Group Inc.<br />
  43. References<br />We’re Recruiting<br />kaChing’s blog<br />kawala<br />Deployment Infrastructure<br />Extreme Testing<br />Writing Testable Code<br />ZooKeeper<br />Eric Ries<br />Pascal-Louis Perez, kaChing Group Inc.<br />