0
Continuous Delivery
in a Complex S.O.A.
Richard Lennox
Senior Architect, Skyscanner
www.skyscanner.net
@richardlennox
www....
Skyscanner
Why Continuous Delivery?
Continuous Delivery
“How long would it take your organisation to deploy a
change that involved just a single line of code?...
Goals of CD
Quality Cycle Time
Core Practices
 Continuous Integration
 Automated Testing
 [Automated] Configuration Management
 Deployment Pipelines
Continuous Integration
Continuous Integration
 Everyone Commits to the Mainline Every
Day
 Automate the Build
 Make Build Self Testing
 Every...
Automated Testing
Build Quality In
“Cease dependence on mass inspection to
achieve quality. Improve the process and build
quality into the p...
Automated Testing
Cost of
Execution
Time
Number of Tests
Unit Tests
Integration Tests
UI Tests
100% Automated
100% Manual
...
Configuration Management
Configuration Management
 Stuff we make is valuable
 Not just the source code :
 Application configuration
 OS setup
...
Deployment Pipeline
Deployment Pipeline
8 Principles / Patterns
 Reliable & Repeatable delivery process
 Automate (almost) everything
 Take the pain early and ...
Service Oriented Architecture
Definition:
- software architecture design pattern
- structured collections of discrete soft...
Vision of Necessary State
Individual Services
Working of own heartbeat
Considerations for CD in SOA
Focus on 5 key areas
- Partitioning the SOA into governed
Deployable Units
- Consistent Deplo...
Deployable Units
 1+ Services to be deployed
 Single, built once Package Comprising:
 Releasable Software
 Tests
 App...
Not a 1:1 ratio?
Service Governance
a.k.a. Making sure people do the right things –
Anne Thomas Manes, OOP2007
 SLAs
 Versioning
 Fault ...
Service Compatibility Governance
Client
(v1)
Client
(v2)
Service
(v1)
Service
(v2)
Deployment
Deployment
Rollback
Rollback
Databases
 Services may have own DB
 Versioned with
 Some DBs act as integration points
 Legacy
 Must be treated as a...
Consistent Deployment Pipelines
 Early Integration not Late Integration
 Consistency, Consistency and Consistency
 Buil...
Just Enough Test Automation
Cost of
Execution
Time
Number of Tests
Unit Tests
Integration Tests
Acceptance Tests
UI Tests
...
Dependency Test Triggers
C D E
B
A
Y
XX
ZI
Real Time Metrics
 Safety Net for canary release
 Coupled with Operational Metrics
 Individual Service defined
 Asynch...
Cascading Healthchecks
1
F G D
E
A
H
C
I
B
2 3
Aligning Delivery Conditions
Culture
 Agile Practices First
 Breaking down Silos
 Getting over the fear of collaboration
 Trunk Based Development o...
Summary
 Architecture has a huge impact on the
ability to Continuously Deliver
 Comes with its own set of challenges and...
Thanks!
Richard Lennox
Senior Architect, Skyscanner
www.skyscanner.net
@richardlennox
www.linkedin.com/in/rlennox
Question...
Upcoming SlideShare
Loading in...5
×

Continuous Delivery in a Complex S.O.A.

647

Published on

Slides from STV Tech Talks (Glasgow, UK) - 27 August 2013
--
Based on the experience of leading the current initiative to move towards Continuous Delivery at Skyscanner, I presented a view on 5 key focus areas that must be considered to make Continuous Delivery an achievable objective within a very complex Service Oriented Architecture.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
647
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
19
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • As Michael has pointed out. Skyscanner – I am sure most of you know Skyscanner – even those few of you that work there… But for those wanting some idea of the scale.:#1 flight search engine across Europe, #3 in the world for traffic. Fastest Growing international travel search site outside of the US.50-75million visits per month40+ languages (some of which are in process of going live) in 30+ local markets and in 70+ currencies200 members of staff globally (closer to 300 now I think) – across 5 offices – Edinburgh, Galsgow, Singapore, Beijing and Miami. +150 by end of the year.Engineering is centred in Scotland, primarily Edinburgh but also Glasgow. Overall 120+ in Development + Test Engineering, 20+ in Service Management (ops). I’ll come on to a few technical implementations.Topic:I am currently leading the current project to bring Continuous Delivery practices to Skyscanner. I am going to talk about the lessons learned so far in doing that, the compromises we’ve made an how it applies to our complex S.O.A. We could talk all day on the topic and as I have an hour (and I am getting back to Edinburgh on the last train) – we are going to skirt some of the detail and keep it high level.Quickly introduce the topic – How many people understand the principles of Continuous Delivery? We’ll do a recap but I’ll be keeping it very brief, and my interpretation of it. First off I’d like to make a statement – IMHO the value of Agile isn’t in the prescription of the SCRUM or XP flavours. Agile development is about sticking to the principles but finding out exactly what the What is Continuous DeliveryWhat is SOAWhat is that at Skyscanner? Why bother ? Perfect Storm (Test team building up, defining release operations; too much pain)5 key challenges and how we are planning to solve itGet business buy in…Deployable Units (Monolithic vs component)Automating TestsService GovernanceLate integrationConsistency
  • 50+ logical components 30+ “physical” services + web apps and Both:Message based (RabbitMQ)Restful Web ServicesMixed Technology Stack.Net (C#)Python JS / CSS / Mobile applicationsHA and Resiliance2 DCs (4 by Q2 next year)
  • Literally playing through the pain…Some of our core applications and services are propped up by levels of effort every 4-6 weeks that they really shouldn’t be. Yes we have a strong SOA, component based architecture but we struggle to release – particualrly now we are at a scale of multiple DCs worldwide; hundreds of virtual machines.
  • Mary & Tom Popendieck, Implementing Lean Software Development
  • Martin Fowler:“Everyone should commit to the mainline at least once every day”Check in daily – checkRuns through builds. The values of the feature branching etc.
  • Detecting when things go wrong – Building Quality in…Unit Testing from the developers
  • Statistician said these words.. Fixing issues when they are created is by far cheaper than finding them at the end.The example is - In building quality at each stage of the German car production line, the quality teams at the end of the process had nothing to do. And if there’s one thing the Germans do well its penality taking and building cars. Essentially the same for Software Engineering.And the way we do that is through Automated Testing – at multiple levels right around the entire system.
  • Version everythingWhere ever possible we should be automating that. Build Once – Deploy manyIntroduce the concept of a deployable unit.Data is a challenge – whether you look at something like DB Deploy.
  • Ubiquitous Pipeline Image for Continuous Delivery Presentations….What does it mean?A deployment (or Build) pipeline is an automated manifestation of the process for getting the software out of versions control and in front of users.
  • Particularlly simple example and know where near good enough.
  • Loose Coupling between product systems enables small batchesServices have a single Responsibility – yes.Not the same as a deployable unit…
  • Pretty simple statement – a key – value.Own heartbeat…I think the CD guys got it wrong when they talked about “Fast” – I believe in Efficient. Continuous Delviery is about having the mechanisms to “Once a product developer realises that small batches are desirable, the start adopting product architectures that permit work to flow in small, decoupled batches.”Principles of product Development Flow, Don Reinertson.
  • Plannign carefullyIf 2 services move in direct lock step, then you don’t need to explode them into indeividual services. You do need to be very careful of integration testing nightmare of O to the NDelivering the elements
  • MakingMassively Important for Services to be governed correctly. For many SOAs an Enterprise Service Bus is uesed to enforce and delvier the governance. That is one approach. But in its essence it is an abstract se
  • Non-breaking changes are required. Any breaking changes are required to be Versioned and delivered.--Server Backward Compatability & Client Forward Compatibilty- New version of service must support v1 of client written to old interfaceClient continues to function against v2 of a service written to the new interdace.Here is the extra bits in this diagram. Took me a long time to get my head around this.Client Backward CompatibilityNew version of client – should continue to support v1 of Service as well as v2Server Forward compatibilityExisting Services must work with newer clients that written to newer interface.A non breaking change to this might be the addition of some optional parameters to a service call or some extra fields in the JSON response. This should be handled gracefully by the client with no changes required. Having the service also ignore unnecessary parameters rather than fail also delivers this level of compatibility.Or in a breaking change then the client must be aware that All of this supports releasing canaries and test elements.
  • Late integration vs. Early integration
  • Revisit the test pyramid.As part of our exapnded growth – massively increased our test automation capabilities in house. Those guys are delivering a shift towards the elements.Using a variety of integration tests against services dricectly and BDD techniques.They are applying it within the areas of expertise.Employ a Testing StrategyIdentify and Prioritise based on RiskDecide What Actions To TakeTests = Executable SpecificationsIncreasing ConfidenceAllows manual testing to be focussed on Exploratory rather than the mundane strategic elements
  • Initially each Service is aware of the services on which it depends on and healthchecks are necessary. Looking at this diagram there are 3 healthchecks running at the outer layer. Each in turn bases it health, not just on itself but also the state of the healthcheck below.This takes effort to maintain on behalf of the component.sIn some SOA’s the ESB will take care of this. But an ESB is a large wieldy piece of kit – and not essentail for SOA.
  • Me – Well I am me - WTF is an architect – well I would summarise it – a technical leads who has a broader field of vision when it comes to the application both in the systems space and in time. See byond the delviery of the current task and sets out the standards. Skyscanner – Who knows about Skyscanner? We are based in Edinburgh, have new glasgow office (+ singapore, Beijing and Miami) – all engineering is done in Scotland. – and yes we are hiring good engineers. Uve been there almost 4 years and therefore am a veteran. Topic:I am currently leading an underway project to bring Continuous Delivery practices to Skyscanner. I am going to talk about the lessons learned so far in doing that, the compromises we’ve made an how it applies to our complex S.O.A. We could talk all day on the topic and as I have an hour (and I am getting back to Edinburgh on the last train) – we are going to skirt some of the detail and keep it high level.Quickly introduce the topic – How many people understand the principles of Continuous Delivery? We’ll do a recap but I’ll be keeping it very brief, and my interpretation of it. First off I’d like to make a statement – IMHO the value of Agile isn’t in the prescription of the SCRUM or XP flavours. Agile development is about sticking to the principles but finding out exactly what the What is Continuous DeliveryWhat is SOAWhat is that at Skyscanner? Why bother ? Perfect Storm (Test team building up, defining release operations; too much pain)5 key challenges and how we are planning to solve itGet business buy in…Deployable Units (Monolithic vs component)Automating TestsService GovernanceLate integrationConsistency
  • Transcript of "Continuous Delivery in a Complex S.O.A."

    1. 1. Continuous Delivery in a Complex S.O.A. Richard Lennox Senior Architect, Skyscanner www.skyscanner.net @richardlennox www.linkedin.com/in/rlennox
    2. 2. Skyscanner
    3. 3. Why Continuous Delivery?
    4. 4. Continuous Delivery “How long would it take your organisation to deploy a change that involved just a single line of code?” Do you do this on a repeatable, reliable basis?” …is a set of practices and principles aimed at building, testing and releasing software reliably and repeatably at a necessary frequency. Keeping systems production-ready throughout development, so that they can be released to users at any time
    5. 5. Goals of CD Quality Cycle Time
    6. 6. Core Practices  Continuous Integration  Automated Testing  [Automated] Configuration Management  Deployment Pipelines
    7. 7. Continuous Integration
    8. 8. Continuous Integration  Everyone Commits to the Mainline Every Day  Automate the Build  Make Build Self Testing  Every Commit triggers build on Integration server  Fail Fast, Feedback Faster http://martinfowler.com/articles/continuousIntegration.html
    9. 9. Automated Testing
    10. 10. Build Quality In “Cease dependence on mass inspection to achieve quality. Improve the process and build quality into the product in the first place” W. Edwards Deming
    11. 11. Automated Testing Cost of Execution Time Number of Tests Unit Tests Integration Tests UI Tests 100% Automated 100% Manual Acceptance Tests
    12. 12. Configuration Management
    13. 13. Configuration Management  Stuff we make is valuable  Not just the source code :  Application configuration  OS setup  Machine Configuration  Tests  Documentation  Where possible should live with the code
    14. 14. Deployment Pipeline
    15. 15. Deployment Pipeline
    16. 16. 8 Principles / Patterns  Reliable & Repeatable delivery process  Automate (almost) everything  Take the pain early and more often  Source control is king  Done or Done, Done = Released,  Build quality in  Everybody is responsible  Continuous improvement
    17. 17. Service Oriented Architecture Definition: - software architecture design pattern - structured collections of discrete software modules (the services) - collectively provide the complete functionality of a large software application
    18. 18. Vision of Necessary State Individual Services Working of own heartbeat
    19. 19. Considerations for CD in SOA Focus on 5 key areas - Partitioning the SOA into governed Deployable Units - Consistent Deployment Pipelines - Just Enough Test Automation Coverage - Real Time Metrics - Aligning the Delivery Conditions
    20. 20. Deployable Units  1+ Services to be deployed  Single, built once Package Comprising:  Releasable Software  Tests  Application Configuration  “Install” / Rollback Scripts  Assets  One mainline (Trunk) per Deployable Unit  Decoupled Deployments and Rollbacks  Fits naturally with S.O.A.  But not necessarily 1:1 ratio
    21. 21. Not a 1:1 ratio?
    22. 22. Service Governance a.k.a. Making sure people do the right things – Anne Thomas Manes, OOP2007  SLAs  Versioning  Fault Tolerance and Resilient Engineering
    23. 23. Service Compatibility Governance Client (v1) Client (v2) Service (v1) Service (v2) Deployment Deployment Rollback Rollback
    24. 24. Databases  Services may have own DB  Versioned with  Some DBs act as integration points  Legacy  Must be treated as any other Service  Same levels of governance,  Same SLAs  Same Tolerance to failures
    25. 25. Consistent Deployment Pipelines  Early Integration not Late Integration  Consistency, Consistency and Consistency  Build Scripts  Tools of choice,  Test Frameworks and Usage  Everything where possible  Canary Release Patterns  Aligning to a single flow for all components
    26. 26. Just Enough Test Automation Cost of Execution Time Number of Tests Unit Tests Integration Tests Acceptance Tests UI Tests 100% Automated 100% Manual
    27. 27. Dependency Test Triggers C D E B A Y XX ZI
    28. 28. Real Time Metrics  Safety Net for canary release  Coupled with Operational Metrics  Individual Service defined  Asynchronous Data Loaders – throughput  Failing Fast and Hard
    29. 29. Cascading Healthchecks 1 F G D E A H C I B 2 3
    30. 30. Aligning Delivery Conditions
    31. 31. Culture  Agile Practices First  Breaking down Silos  Getting over the fear of collaboration  Trunk Based Development over Feature Branching  Discipline in Continuous Integration  Dealing with breaks
    32. 32. Summary  Architecture has a huge impact on the ability to Continuously Deliver  Comes with its own set of challenges and  It is the right thing for modern web-based businesses.
    33. 33. Thanks! Richard Lennox Senior Architect, Skyscanner www.skyscanner.net @richardlennox www.linkedin.com/in/rlennox Questions?
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×