SlideShare a Scribd company logo
1 of 44
Applied Lean Startup Ideas: Continuous Deployment at kaChing & #leanstartup
Twenty Deployments a Day Keeps the Nasty Bugs Away Pascal-Louis Perez, kaChing Group Inc.
“Connect investors with outstandinginvestment managers” Pascal-Louis Perez, kaChing Group Inc.
Release is a Marketing ConcernReducing Code Inventory is what Engineering Focuses On Pascal-Louis Perez, kaChing Group Inc.
Lean Startup at kaChing “Human institution designed to create something new under conditions of extreme uncertainty” – Eric Ries Methodical approach to eliminating waste No Quality Assurance engineers, no Operations team Empower everybody to drive their own projects Pascal-Louis Perez, kaChing Group Inc.
What is Continuous Deployment? Continuous, successful and repeatable methodology to deploying code Automates every steps of taking checked in code and making it run on production servers, in front of customers Pascal-Louis Perez, kaChing Group Inc.
True Story Investment managers calls, comments about unintuitive trading flow. Improvements are made, and deployed within the next 20 minutes. We call him back “What do you think of the improvement?” Pascal-Louis Perez, kaChing Group Inc.
Benefits of Continuous Deployment It allows quick iterations Obsoletes processes, e.g. “cutting a release” Reduces risk Everyone is aware of production No one throws code over the wall Exposes 24x7 operational requirements Trunk stable Pascal-Louis Perez, kaChing Group Inc.
ContinuousDeployment Immune System Making Continuous Deployment a Reality Pascal-Louis Perez, kaChing Group Inc. Culture Test Driven Development
#architecture Pascal-Louis Perez, kaChing Group Inc.
Our architecture Service oriented system Vertical sharding Everything uses the same platform kawala Coordination using ZooKeeper Data interchange using JSON and Protobufs Pascal-Louis Perez, kaChing Group Inc.
Typical Stack Clustered services, multiple instances Replicated databases (e.g. MySQL or NoSQL) Caching (e.g. memcached) Denormalized data (e.g. Voldemort) Pascal-Louis Perez, kaChing Group Inc.
kawala Command pattern. A command is a query Dynamic Inversion of Control Queries produce a value Pascal-Louis Perez, kaChing Group Inc.
Sample Query Pascal-Louis Perez, kaChing Group Inc. class GetUser(Id<User> userId) {    @Inject UserRepository repository;    public User process() {        return repository.load(userId);    }}
What We do with Queries Pascal-Louis Perez, kaChing Group Inc. Persist Queries RPC Command Line Tools Serve BLOBS CPS Render Pages Push into MQ
Lifecycle 1. Commit 2. Testing 3. Deployment 4. Production Validation 5. Lifetime Monitoring
Development Trunk stable Small, frequent commits Unreleased features live behind experiments Forward & Backward compatibility Responsive code review Pair difficult problems Trivial rollbacks
#testing
Testing Philosophy Only automated testing matters If it isn’t tested, it isn’t finished or correct Write testable code Embrace abstractions Testing is cross functional, we all own quality
Benefits of TDD It allows quick iterations It empowers engineers to change anything, and as such helps in scaling the team It is more cost effective than debugging It obsoletes the need for functional QA It facilitates continuous refactoring, allowing the code to get better with age It attracts the right kind of engineers Pascal-Louis Perez, kaChing Group Inc.
Types of Testing Unit Testing - does the code work? Integration Testing - does the code work together? Regression Testing - learn from your mistakes Frontend Testing - a whole different ballgame
Defensive Testing Nightmare scenarios A common conversation at lunch: Alice: What would happen if X blew up? Bob: uh... the site would go down. Fix it, test for it
Implementation Frameworks: junit, dbunit CI: Hudson Frontend: Selenium
Production Process, Infrastructure, Deployment, Monitoring
Process Everyone: writes code and tests, releases to production, adds monitoring Specialists help out when needed New hires push code the first day Problems, issues, errors, bugs, oversights...
Stability Cluster everything As little global state as possible Maintain global state through ZooKeeper Monitor everything
ZooKeeper Reliable Distributed Synchronization For: Service Discovery Service Status Coarse locking
Life of a Deployment Pascal-Louis Perez, kaChing Group Inc. DM PM PM PM ZK
Life of a Deployment Pascal-Louis Perez, kaChing Group Inc. DM PM PM PM ZK
Life of a Deployment Pascal-Louis Perez, kaChing Group Inc. DM PM PM ZK
Life of a Deployment Pascal-Louis Perez, kaChing Group Inc. DM PM PM PM ZK
Life of a Deployment Pascal-Louis Perez, kaChing Group Inc. DM PM PM PM ZK
Deployment Exponentially increasing group sizes Increased monitoring during deployment Self Test Automated rollbacks
Implementation Building: ant Deploying: rpm, yum Monitoring: nagios, jcollectd, RRDtool, jmx Custom automation
#monitoring
Monitoring Philosophy Prefer business metrics Monitor statistical deviations not absolute values Automatically annotate graphs
Monitoring Errors False negatives - errors of omission False positives - errors of implementation
End to End Monitoring Run Selenium on production Accessibility, Speed Ad-hoc Keynote - customized for our flows Must control data creation, analytics impact
#future-developments Pascal-Louis Perez, kaChing Group Inc.
Quarantining Isolate new releases Flexible partitioning of requests Gradually shift load to fresh services Pascal-Louis Perez, kaChing Group Inc.
Describing Infrastructure Many moving parts nagios, collectd, backups, services, databases, … Consistency is key Adding new tools should be easy and thorough Standardize best practices Pascal-Louis Perez, kaChing Group Inc.
What we Covered Today Lean Startup and Continuous Deployment Anatomy of a Release Commit, Testing, Deployment, Monitoring Future Developments Quarantining, DSL for Infrastructure Pascal-Louis Perez, kaChing Group Inc.
References We’re Recruiting jobs@kaching.com kaChing’s blog http://eng.kaching.com kawalahttp://bit.ly/kawala Deployment Infrastructure http://eng.kaching.com/2010/05/deployment-infrastructure-for.html Extreme Testing http://bit.ly/9bOFaA Writing Testable Codehttp://googletesting.blogspot.com/2008/08/by-miko-hevery-so-you-decided-to.html ZooKeeperhttp://bit.ly/kc-zookeeper Eric Rieshttp://startuplessonslearned.com Pascal-Louis Perez, kaChing Group Inc.

More Related Content

More from Pascal-Louis Perez

Products’ Love Story with Biz
Products’ Love Story with BizProducts’ Love Story with Biz
Products’ Love Story with BizPascal-Louis Perez
 
How to Send a Receipt, Topics in Concurrency and Distributed Systems
How to Send a Receipt, Topics in Concurrency and Distributed SystemsHow to Send a Receipt, Topics in Concurrency and Distributed Systems
How to Send a Receipt, Topics in Concurrency and Distributed SystemsPascal-Louis Perez
 
Developing an Immune System — The Hard and Soft Skills required to avoid Outages
Developing an Immune System — The Hard and Soft Skills required to avoid OutagesDeveloping an Immune System — The Hard and Soft Skills required to avoid Outages
Developing an Immune System — The Hard and Soft Skills required to avoid OutagesPascal-Louis Perez
 
SLL Conf - Continuous Deployment
SLL Conf - Continuous DeploymentSLL Conf - Continuous Deployment
SLL Conf - Continuous DeploymentPascal-Louis Perez
 
Alchemist Startup Primer - Lean Development Practices
Alchemist Startup Primer - Lean Development PracticesAlchemist Startup Primer - Lean Development Practices
Alchemist Startup Primer - Lean Development PracticesPascal-Louis Perez
 
Applying Compiler Techniques to Iterate At Blazing Speed
Applying Compiler Techniques to Iterate At Blazing SpeedApplying Compiler Techniques to Iterate At Blazing Speed
Applying Compiler Techniques to Iterate At Blazing SpeedPascal-Louis Perez
 
Iterate Like a Whirling Dervish
Iterate Like a Whirling DervishIterate Like a Whirling Dervish
Iterate Like a Whirling DervishPascal-Louis Perez
 
Xignite's Dedicate kaChing Api
Xignite's Dedicate kaChing ApiXignite's Dedicate kaChing Api
Xignite's Dedicate kaChing ApiPascal-Louis Perez
 
Add (Syntactic) Sugar To Your Java
Add (Syntactic) Sugar To Your JavaAdd (Syntactic) Sugar To Your Java
Add (Syntactic) Sugar To Your JavaPascal-Louis Perez
 

More from Pascal-Louis Perez (15)

Fuchsia RFCs
Fuchsia RFCsFuchsia RFCs
Fuchsia RFCs
 
Products’ Love Story with Biz
Products’ Love Story with BizProducts’ Love Story with Biz
Products’ Love Story with Biz
 
How to Send a Receipt, Topics in Concurrency and Distributed Systems
How to Send a Receipt, Topics in Concurrency and Distributed SystemsHow to Send a Receipt, Topics in Concurrency and Distributed Systems
How to Send a Receipt, Topics in Concurrency and Distributed Systems
 
Corporate Finance Primer
Corporate Finance PrimerCorporate Finance Primer
Corporate Finance Primer
 
Developing an Immune System — The Hard and Soft Skills required to avoid Outages
Developing an Immune System — The Hard and Soft Skills required to avoid OutagesDeveloping an Immune System — The Hard and Soft Skills required to avoid Outages
Developing an Immune System — The Hard and Soft Skills required to avoid Outages
 
SLL Conf - Continuous Deployment
SLL Conf - Continuous DeploymentSLL Conf - Continuous Deployment
SLL Conf - Continuous Deployment
 
Alchemist Startup Primer - Lean Development Practices
Alchemist Startup Primer - Lean Development PracticesAlchemist Startup Primer - Lean Development Practices
Alchemist Startup Primer - Lean Development Practices
 
Database compatibility
Database compatibilityDatabase compatibility
Database compatibility
 
Applying Compiler Techniques to Iterate At Blazing Speed
Applying Compiler Techniques to Iterate At Blazing SpeedApplying Compiler Techniques to Iterate At Blazing Speed
Applying Compiler Techniques to Iterate At Blazing Speed
 
Iterate Like a Whirling Dervish
Iterate Like a Whirling DervishIterate Like a Whirling Dervish
Iterate Like a Whirling Dervish
 
Extreme Testing at kaChing
Extreme Testing at kaChingExtreme Testing at kaChing
Extreme Testing at kaChing
 
Type Checking JavaScript
Type Checking JavaScriptType Checking JavaScript
Type Checking JavaScript
 
Xignite's Dedicate kaChing Api
Xignite's Dedicate kaChing ApiXignite's Dedicate kaChing Api
Xignite's Dedicate kaChing Api
 
Add (Syntactic) Sugar To Your Java
Add (Syntactic) Sugar To Your JavaAdd (Syntactic) Sugar To Your Java
Add (Syntactic) Sugar To Your Java
 
kaChing's API garage event
kaChing's API garage eventkaChing's API garage event
kaChing's API garage event
 

Recently uploaded

Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...Rick Flair
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality AssuranceInflectra
 
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...AliaaTarek5
 
Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Hiroshi SHIBATA
 
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentEmixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentPim van der Noll
 
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfSo einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfpanagenda
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsSergiu Bodiu
 
Take control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteTake control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteDianaGray10
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxLoriGlavin3
 
TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024Lonnie McRorey
 
Scale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL RouterScale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL RouterMydbops
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxLoriGlavin3
 
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxMerck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxLoriGlavin3
 
The Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsThe Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsPixlogix Infotech
 
Generative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information DevelopersGenerative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information DevelopersRaghuram Pandurangan
 
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024BookNet Canada
 
DSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningDSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningLars Bell
 
Time Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directionsTime Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directionsNathaniel Shimoni
 
What is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfWhat is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfMounikaPolabathina
 
The State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxThe State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxLoriGlavin3
 

Recently uploaded (20)

Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
 
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...
 
Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024
 
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentEmixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
 
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfSo einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platforms
 
Take control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteTake control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test Suite
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
 
TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024
 
Scale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL RouterScale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL Router
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
 
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxMerck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
 
The Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsThe Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and Cons
 
Generative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information DevelopersGenerative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information Developers
 
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
 
DSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningDSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine Tuning
 
Time Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directionsTime Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directions
 
What is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfWhat is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdf
 
The State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxThe State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptx
 

Applied Lean Startup Ideas: Continuous Deployment at kaChing

  • 1. Applied Lean Startup Ideas: Continuous Deployment at kaChing & #leanstartup
  • 2. Twenty Deployments a Day Keeps the Nasty Bugs Away Pascal-Louis Perez, kaChing Group Inc.
  • 3. “Connect investors with outstandinginvestment managers” Pascal-Louis Perez, kaChing Group Inc.
  • 4. Release is a Marketing ConcernReducing Code Inventory is what Engineering Focuses On Pascal-Louis Perez, kaChing Group Inc.
  • 5. Lean Startup at kaChing “Human institution designed to create something new under conditions of extreme uncertainty” – Eric Ries Methodical approach to eliminating waste No Quality Assurance engineers, no Operations team Empower everybody to drive their own projects Pascal-Louis Perez, kaChing Group Inc.
  • 6. What is Continuous Deployment? Continuous, successful and repeatable methodology to deploying code Automates every steps of taking checked in code and making it run on production servers, in front of customers Pascal-Louis Perez, kaChing Group Inc.
  • 7. True Story Investment managers calls, comments about unintuitive trading flow. Improvements are made, and deployed within the next 20 minutes. We call him back “What do you think of the improvement?” Pascal-Louis Perez, kaChing Group Inc.
  • 8. Benefits of Continuous Deployment It allows quick iterations Obsoletes processes, e.g. “cutting a release” Reduces risk Everyone is aware of production No one throws code over the wall Exposes 24x7 operational requirements Trunk stable Pascal-Louis Perez, kaChing Group Inc.
  • 9. ContinuousDeployment Immune System Making Continuous Deployment a Reality Pascal-Louis Perez, kaChing Group Inc. Culture Test Driven Development
  • 10. #architecture Pascal-Louis Perez, kaChing Group Inc.
  • 11. Our architecture Service oriented system Vertical sharding Everything uses the same platform kawala Coordination using ZooKeeper Data interchange using JSON and Protobufs Pascal-Louis Perez, kaChing Group Inc.
  • 12. Typical Stack Clustered services, multiple instances Replicated databases (e.g. MySQL or NoSQL) Caching (e.g. memcached) Denormalized data (e.g. Voldemort) Pascal-Louis Perez, kaChing Group Inc.
  • 13. kawala Command pattern. A command is a query Dynamic Inversion of Control Queries produce a value Pascal-Louis Perez, kaChing Group Inc.
  • 14. Sample Query Pascal-Louis Perez, kaChing Group Inc. class GetUser(Id<User> userId) { @Inject UserRepository repository; public User process() { return repository.load(userId); }}
  • 15. What We do with Queries Pascal-Louis Perez, kaChing Group Inc. Persist Queries RPC Command Line Tools Serve BLOBS CPS Render Pages Push into MQ
  • 16. Lifecycle 1. Commit 2. Testing 3. Deployment 4. Production Validation 5. Lifetime Monitoring
  • 17. Development Trunk stable Small, frequent commits Unreleased features live behind experiments Forward & Backward compatibility Responsive code review Pair difficult problems Trivial rollbacks
  • 19. Testing Philosophy Only automated testing matters If it isn’t tested, it isn’t finished or correct Write testable code Embrace abstractions Testing is cross functional, we all own quality
  • 20. Benefits of TDD It allows quick iterations It empowers engineers to change anything, and as such helps in scaling the team It is more cost effective than debugging It obsoletes the need for functional QA It facilitates continuous refactoring, allowing the code to get better with age It attracts the right kind of engineers Pascal-Louis Perez, kaChing Group Inc.
  • 21. Types of Testing Unit Testing - does the code work? Integration Testing - does the code work together? Regression Testing - learn from your mistakes Frontend Testing - a whole different ballgame
  • 22. Defensive Testing Nightmare scenarios A common conversation at lunch: Alice: What would happen if X blew up? Bob: uh... the site would go down. Fix it, test for it
  • 23. Implementation Frameworks: junit, dbunit CI: Hudson Frontend: Selenium
  • 24. Production Process, Infrastructure, Deployment, Monitoring
  • 25. Process Everyone: writes code and tests, releases to production, adds monitoring Specialists help out when needed New hires push code the first day Problems, issues, errors, bugs, oversights...
  • 26. Stability Cluster everything As little global state as possible Maintain global state through ZooKeeper Monitor everything
  • 27. ZooKeeper Reliable Distributed Synchronization For: Service Discovery Service Status Coarse locking
  • 28. Life of a Deployment Pascal-Louis Perez, kaChing Group Inc. DM PM PM PM ZK
  • 29. Life of a Deployment Pascal-Louis Perez, kaChing Group Inc. DM PM PM PM ZK
  • 30. Life of a Deployment Pascal-Louis Perez, kaChing Group Inc. DM PM PM ZK
  • 31. Life of a Deployment Pascal-Louis Perez, kaChing Group Inc. DM PM PM PM ZK
  • 32. Life of a Deployment Pascal-Louis Perez, kaChing Group Inc. DM PM PM PM ZK
  • 33. Deployment Exponentially increasing group sizes Increased monitoring during deployment Self Test Automated rollbacks
  • 34. Implementation Building: ant Deploying: rpm, yum Monitoring: nagios, jcollectd, RRDtool, jmx Custom automation
  • 35.
  • 37. Monitoring Philosophy Prefer business metrics Monitor statistical deviations not absolute values Automatically annotate graphs
  • 38. Monitoring Errors False negatives - errors of omission False positives - errors of implementation
  • 39. End to End Monitoring Run Selenium on production Accessibility, Speed Ad-hoc Keynote - customized for our flows Must control data creation, analytics impact
  • 41. Quarantining Isolate new releases Flexible partitioning of requests Gradually shift load to fresh services Pascal-Louis Perez, kaChing Group Inc.
  • 42. Describing Infrastructure Many moving parts nagios, collectd, backups, services, databases, … Consistency is key Adding new tools should be easy and thorough Standardize best practices Pascal-Louis Perez, kaChing Group Inc.
  • 43. What we Covered Today Lean Startup and Continuous Deployment Anatomy of a Release Commit, Testing, Deployment, Monitoring Future Developments Quarantining, DSL for Infrastructure Pascal-Louis Perez, kaChing Group Inc.
  • 44. References We’re Recruiting jobs@kaching.com kaChing’s blog http://eng.kaching.com kawalahttp://bit.ly/kawala Deployment Infrastructure http://eng.kaching.com/2010/05/deployment-infrastructure-for.html Extreme Testing http://bit.ly/9bOFaA Writing Testable Codehttp://googletesting.blogspot.com/2008/08/by-miko-hevery-so-you-decided-to.html ZooKeeperhttp://bit.ly/kc-zookeeper Eric Rieshttp://startuplessonslearned.com Pascal-Louis Perez, kaChing Group Inc.

Editor's Notes

  1. Thanks to SDForum, thanks to LinkedIn for hostingToday, talk about applied lean startup and specifically about continuous deployment at kaChingDavid and I are presenting the efforts of a whole team.Tweet, share, shout, ask questionsBefore we start, let me ask you a few questions:How long from writing code to having it in production?How much time do spend on “releasing” every month?How long does deprecated code stay around?How often do you refactor code?New hire code to production?How long from product request to code in production, in front of alpha/beta testers?
  2. Let me give a some context about what kaChing is.Our goal is to connect investors with outstanding investment managers.Schwab created Schawb One Source which is THE marketplace for mutual funds.We want to create the equivalent for individually managed accounts, essentially bringing wealth management to retail.On kaChing, transparency is the norm. You can know everything about the investment manager you are entrusting. His portfolio holdings, his past transaction history, his rational, philosophy. And we use all this data to objectively vet managers.In 6 months, we’ve attracted $11M and are growing very fast.We’re an SEC regulated company, en route to being FINRA regulated as a broker/dealer.
  3. QA and operations engineer are typically less compensated than software engineers or architects. How can we be saving by having more expensive people do the work?Also explain what our view on QA is: two functions. Check the spec was translated to a working product correctly; Check that the product works. The first part is inherently human, the latter must be automated to eliminate waste.
  4. CD’s benefits are twofold: business/lean startup and technology related.Continuous deployment is essential to driving engineering culture and enabling quick iteration cycles.Many processes which are typically seen in engineering organizing, cutting a release weekly for instance, are not needed anymore. Reduces a lot of waste, that is work not directly helping with the learning experience of creating something newWe are a financial company. We deal with people’s money. Not being able to communicate with our prime broker is a no no. Work in small batches, reduce the possibility of integration problems, and rollback as well as narrow down quickly if they ever occur.From a technology stand point, engineers are acutely aware of how things run. You are on the hook for bad code, no throwing over the wall to QA organization.Everybody needs to know about how a 24x7 site is ran.Trunk stable, we’ll talk about that in a minute
  5. This is a very high level survey, the goal is to convey the necessary things to understand how our system works in the context of CD not how kaChing is built.JSON and Protobufs
  6. -Scale the team: add features without adding engineers to support and maintain these features. This requires “production tests” a.k.a. monitoring.-Debugging: debugging live servers in a production environment!-At kaChing, we’ve never spent $1 on QA-Not all engineers are excited by a test-driven environment. Some are happy with “tests” taking “only” 5 hours to run.