More Related Content


Architecting for Continuous Delivery

  1. Architecting for Continuous Delivery Russ Barnett, Chief Architect John Esser, Director of Engineering Productivity
  2. News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month Watch the video with slide synchronization on! /ancestry-SOA-continuous-delivery
  3. Presented at QCon San Francisco Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide
  4. Background 2
  5. Growth at • Subscribers have doubled in the past 3 years – (~1M to > 2M) • Page views have doubled in the past 3 years – (~25M/day to ~50M/day) • Development head count has tripled – (100 to 300) • Feature throughput has dramatically increased 3
  6. Continuous Delivery is consistently and reliably releasing business value increments fast through automated build, test, configuration and deployment. What is the value of going fast? A LOT!
  7. Evolution to Continuous Delivery Agile Boot Up (Scrum) Enterprise Agile Framework Agile Architecture Standards Continuous Delivery Adoption Future – “Agile v2” Two year period (2010 – present)
  8. Typical Impediments to Continuous Delivery • Cultural • Technical practices • Quality ownership • Infrastructure • Architectural
  9. Limiting Factors • Pipeline serialized at integration – Errors that occurred in this stage stalled the pipeline • Stalls in integration induced additional problems • Increasing frequency of stalls – As number of development teams grew, frequency of stalls increased 7 Integration Source Control Build Unit Testing System Testing Deployment Team A Team B Team C error causes stall
  10. Everything was coupled! (Aka, large batch size) (It became known as the “big blob!”) 8
  11. 9
  12. Little’s Law 𝑊𝑎𝑖𝑡 𝑇𝑖𝑚𝑒 = 𝑄𝑢𝑒𝑢𝑒 𝑠𝑖𝑧𝑒 𝑃𝑟𝑜𝑐𝑒𝑠𝑠𝑖𝑛𝑔 𝑅𝑎𝑡𝑒 𝑄𝑢𝑒𝑢𝑒 𝑠𝑖𝑧𝑒 ∝ 𝐵𝑎𝑡𝑐ℎ 𝑠𝑖𝑧𝑒 We can reduce wait time (cycle time) by reducing batch size without changing demand or capacity. 10
  13. Problems with Large Batches • Increases cycle time • Increases variability non-linearly as 2n • Increases risk • Reduces efficiency • Limited by its worst element. from Principles of Product Development Flow, Don Reinertsen 11
  14. Answer? Utilize small batches. 12
  15. Fluidity Principle Loose coupling between product systems enables small batches “Once a product developer realizes that small batches are desirable, they start adopting product architectures that permit work to flow in small, decoupled batches.” from Principles of Product Development Flow, Don Reinertsen 13
  16. Fluidity Principle Loose coupling between product systems enables small batches “Once a product developer realizes that small batches are desirable, they start adopting product architectures that permit work to flow in small, decoupled batches.” from Principles of Product Development Flow, Don Reinertsen 14
  17. Creating an Architecture for Agility 15
  18. Architectural Impediments • Cross-Component Coupling – Creates groups of systems that must be deployed together • Insufficient Rollback Capability – Causes teams to resort to cascading rollback • Poor Testing and Monitoring – Requires a long testing period – Lengthens feedback cycle – Allows quality problems to escape to and affect customers 16
  19. Architectural Methods for Removing Impediments • Partition into small single-responsibility components “There should only be one reason for a [component] to change” - Robert Martin • Decouple deployment of components – Separately deploy components – Remove order dependent deployment • Support Independent Rollback – Enforce strict backward and forward compatibility 17
  20. Backward and Forward Compatibility Client Version 1 Service Version 1 Service Version 2 Client Version 2 Deployment Rollback Deployment Rollback Backward Compatibility Backward Compatibility Forward Compatibility Forward Compatibility • Server Backward Compatibility – Newer servers work with clients written to old interface • Server Forward Compatibility – Existing servers work with clients written to newer interface – Supports early client deployment • Client Backward Compatibility – Newer clients work with servers that implement old interface – Supports server rollback • Client Forward Compatibility – Old clients work with servers that implement new interface
  21. Enforcing Decoupled Components • Implementing standards is insufficient – Independent deployment forces some decoupling – High rate of deployment issues indicate remaining coupling • Improve integration testing – Verify backward and forward compatibility – Identify breaking changes quickly – Make writing integration tests easier 19
  22. Improving Interface Verification 20 Server Client In Process Client/Server Server Client • Remember when you could run your entire application in one process? • How do we get better interface verification with services? In Process Thrift SOAP/WSDL REST Verifiable Decoupled Google Protocol Buffers
  23. Interface Verification using Proxies and Stubs 21 Stub Proxy Client Server Client/Server with Proxy/Stub • Verifies interface at compile time • Isolates code from versioning issues • Easier to provide mock implementations • Can test backward and forward compatibility
  24. Managing a Complex Network of Services • Ancestry Scale – About 40 different teams – Over 300 separate application or service systems – Stack is 5+ levels deep • Historical Diagnostics – Presented a client centric or top down view – Insufficient for identifying problems in a network of services • Solution: Deep Status Check – Components provide dynamic status information for each client and dependency – Report traverses dependencies up to a given depth 22
  25. Ancestry Deep Status Check 23 Service 1 Service A Service C Service B App 2 Service 2 GetStatus Monitor App 1 Unidentified Client Unregistered Client Connection Error Database File System Level 1 Level 2 Level 3 Level 4
  26. < 2 s 95% < 4 s fail 99.9% SLA Service Level Agreements • Understand Business Expectations – Each application and service establishes a contract with each client specifying the expected performance characteristics 24 • Methodology – Use percentile buckets rather than an average – Performance is a component of availability
  27. Verify Client Identity Check Circuit Breaker Track Each Incoming Request Report Compliance Incoming Requests Check Circuit Breaker Provide Fallback Mechanism Track Each Outgoing Request Report Compliance Outgoing Requests Service Level Agreement System • Compare incoming and outgoing SLA compliance Service
  28. Conclusion • Architecture affects agility and continuous delivery capability as much or more than other factors. • Process and tool improvements alone are insufficient. • Good architecture techniques enable effective continuous delivery at large scale. – Partition to single-responsibility components. – Decouple deployment – Support independent rollback – Improve testing and monitoring infrastructure 26
  29. • Questions? Contact info: Ancestry is hiring in San Francisco and Utah 27