Continuous Delivery
Practices and real experiences
I’m Eduardo Ferro
VP Platform Engineering Nextail
Available at:
@eferro
https://www.eferro.net
Hi!
2
Agile & DevOps
What the data says
1
3
4
5
Accelerate: The Science of Lean Software and Devops (link) / DORA Report 2019 (link)
6
Accelerate: The Science of Lean Software and Devops (link) / DORA Report 2019 (link)
7
Accelerate: The Science of Lean Software and Devops (link) / DORA Report 2019 (link)
Velocity increase
Deployment Frequency
Myth
8
Stability/Availability decrease
Built-in Quality
9
10
“
11
“Firms with high-performing IT
organizations were twice as likely to exceed
their profitability, market share and
productivity goals”
12
13
14
DORA Report 2019 (link)
15
DORA Report 2019 (link)
16
Agile & DevOps Works!!!
17
Should work, right?2
18
“
According to Gartner 70% of the IT market is focusing
on DevOps and another study found that 88% of
organizations have adopted agile...
Chris Richardson
19
“
According to Gartner 70% of the IT market is focusing
on DevOps and another study found that 88% of
organizations have adopted agile, but only 26% of
organizations have broadly adopted test automations.
Chris Richardson
20
21
“
We are agile, we use Scrum... We don't use any of the
agile technical practices nor do we know what XP is.
22
23
24
Continuous Delivery
¿What? ¿How?
3
25
“
Continuous Delivery is the ability to get changes of
all types—including new features, configuration
changes, bug fixes and experiments
26
https://www.continuousdelivery.com Jez Humble (link)
“
Continuous Delivery is the ability to get changes of
all types—including new features, configuration
changes, bug fixes and experiments—into
production, or into the hands of users
27
https://www.continuousdelivery.com Jez Humble (link)
“
Continuous Delivery is the ability to get changes of
all types—including new features, configuration
changes, bug fixes and experiments—into
production, or into the hands of users, safely and
quickly in a sustainable way.
28
https://www.continuousdelivery.com Jez Humble (link)
29
30
Continuous Delivery
◉ Test automation
◉ Deployment automation
◉ Continuous Integration
◉ Trunk based development
◉ Loosely coupled architectures
◉ Version control
◉ Monitoring / Proactive Notifications
31
Accelerate: The Science of Lean Software and Devops (link) / DORA Report 2019 (link)
Test Automation
◉ XP practice (TDD, Customer tests)
◉ Unit test (fast), integration, acceptance
◉ Other
○ End2end tests
○ Load tests
○ Test in production
32
Deployment automation
◉ Automate deployment pipelines
◉ Remove human errors
◉ Deployment vs Releasing
33
Continuous Integration
◉ XP practice
◉ Trunk branch always prepared for deployment
◉ Every dev merge to trunk every day
◉ High confidence (TDD, Acceptance tests)
◉ Automated build
◉ It's not: using jenkins in Feature Branches
34
Trunk based development
◉ No feature branches or shorter than one day
◉ Small increments
◉ Each increment deployed to production
◉ Require
○ Feature flags
○ Parallel changes
○ Split in small increments (Agile skill)
35
https://trunkbaseddevelopment.com/ (link)
Loosely coupled
architectures
◉ Service oriented architecture
◉ Bounded Context (DDD)
◉ Hexagonal Architecture / Clean Architecture
◉ Each service
○ Individual deployment
○ Can change autonomously
36
Versión Control
◉ Code
◉ Configuration
◉ Infrastructure
◉ Experiments
37
Monitoring and
proactive notifications
◉ Production system Feedback
◉ Architecture validation (fitness functions)
◉ Instrumentation / Observability
38
39
DORA Report 2019 (link)
40
Real Experiences
Agile & Devops
3
41
Nextail 2018-Now
◉ Deployment frequency 1-3/day, cloud, SaaS
◉ On our way to XP & DevOps
◉ TDD (not always)
◉ Pairing/Mob (>30%)
◉ Collective ownership (Product/Code)
◉ Trunk base development (not always)
◉ Hexagonal architecture (for new services)
◉ Splitting the monolith
42
Nextail 2018-Now
◉ Continuous Integration: Partially (on the way)
◉ Continuous Delivery: For all the services
◉ Continuous Deployment: Not yet
◉ Infrastructure as a Code: Terraform
◉ Immutable infrastructure: Yes (aws: eks k8s)
◉ Deployment patterns:
○ Canary releasing, Rolling Deploys
43
TheMotion 2015-2018
◉ Deployment frequency 5-10/day, cloud, SaaS
◉ XP & DevOps practices
◉ TDD
◉ Pairing/Mob (>70%)
◉ Collective ownership (Product/Code)
◉ Trunk base development
◉ Hexagonal architecture
◉ Microservices, Event driven architecture
44
TheMotion 2015-2018
◉ Continuous Integration: For all the services
◉ Continuous Delivery: For all the services
◉ Continuous Deployment: Not for all services
◉ Infrastructure as a Code: Ansible, Terraform
◉ Immutable infrastructure: Yes (aws: ECS)
◉ Deployment patterns:
○ Canary releasing, Rolling Deploys
45
Alea Soluciones 2012-2016
◉ Deployment frequency 5-10/week, on-prem
◉ XP & DevOps practices
◉ TDD
◉ Pairing/Mob (>70%)
◉ Collective ownership (Product/Code)
◉ Trunk base development
◉ Hexagonal architecture
◉ Service-oriented architecture
46
Alea Soluciones 2012-2016
◉ Continuous Integration: For all the services
◉ Continuous Delivery: For all the services
◉ Continuous Deployment: Not for all services
◉ Infrastructure as a Code: Puppet, Ansible
◉ Immutable infrastructure: Partially (docker)
◉ Deployment patterns:
○ Canary releasing, Zero downtime deployment
47
and now, let’s discuss
@eferro
https://www.eferro.net
Thank You!
48

Continuous delivery practices and real experiences

  • 1.
  • 2.
    I’m Eduardo Ferro VPPlatform Engineering Nextail Available at: @eferro https://www.eferro.net Hi! 2
  • 3.
    Agile & DevOps Whatthe data says 1 3
  • 4.
  • 5.
    5 Accelerate: The Scienceof Lean Software and Devops (link) / DORA Report 2019 (link)
  • 6.
    6 Accelerate: The Scienceof Lean Software and Devops (link) / DORA Report 2019 (link)
  • 7.
    7 Accelerate: The Scienceof Lean Software and Devops (link) / DORA Report 2019 (link)
  • 8.
  • 9.
  • 10.
  • 11.
    “ 11 “Firms with high-performingIT organizations were twice as likely to exceed their profitability, market share and productivity goals”
  • 12.
  • 13.
  • 14.
  • 15.
  • 16.
  • 17.
  • 18.
  • 19.
    “ According to Gartner70% of the IT market is focusing on DevOps and another study found that 88% of organizations have adopted agile... Chris Richardson 19
  • 20.
    “ According to Gartner70% of the IT market is focusing on DevOps and another study found that 88% of organizations have adopted agile, but only 26% of organizations have broadly adopted test automations. Chris Richardson 20
  • 21.
  • 22.
    “ We are agile,we use Scrum... We don't use any of the agile technical practices nor do we know what XP is. 22
  • 23.
  • 24.
  • 25.
  • 26.
    “ Continuous Delivery isthe ability to get changes of all types—including new features, configuration changes, bug fixes and experiments 26 https://www.continuousdelivery.com Jez Humble (link)
  • 27.
    “ Continuous Delivery isthe ability to get changes of all types—including new features, configuration changes, bug fixes and experiments—into production, or into the hands of users 27 https://www.continuousdelivery.com Jez Humble (link)
  • 28.
    “ Continuous Delivery isthe ability to get changes of all types—including new features, configuration changes, bug fixes and experiments—into production, or into the hands of users, safely and quickly in a sustainable way. 28 https://www.continuousdelivery.com Jez Humble (link)
  • 29.
  • 30.
  • 31.
    Continuous Delivery ◉ Testautomation ◉ Deployment automation ◉ Continuous Integration ◉ Trunk based development ◉ Loosely coupled architectures ◉ Version control ◉ Monitoring / Proactive Notifications 31 Accelerate: The Science of Lean Software and Devops (link) / DORA Report 2019 (link)
  • 32.
    Test Automation ◉ XPpractice (TDD, Customer tests) ◉ Unit test (fast), integration, acceptance ◉ Other ○ End2end tests ○ Load tests ○ Test in production 32
  • 33.
    Deployment automation ◉ Automatedeployment pipelines ◉ Remove human errors ◉ Deployment vs Releasing 33
  • 34.
    Continuous Integration ◉ XPpractice ◉ Trunk branch always prepared for deployment ◉ Every dev merge to trunk every day ◉ High confidence (TDD, Acceptance tests) ◉ Automated build ◉ It's not: using jenkins in Feature Branches 34
  • 35.
    Trunk based development ◉No feature branches or shorter than one day ◉ Small increments ◉ Each increment deployed to production ◉ Require ○ Feature flags ○ Parallel changes ○ Split in small increments (Agile skill) 35 https://trunkbaseddevelopment.com/ (link)
  • 36.
    Loosely coupled architectures ◉ Serviceoriented architecture ◉ Bounded Context (DDD) ◉ Hexagonal Architecture / Clean Architecture ◉ Each service ○ Individual deployment ○ Can change autonomously 36
  • 37.
    Versión Control ◉ Code ◉Configuration ◉ Infrastructure ◉ Experiments 37
  • 38.
    Monitoring and proactive notifications ◉Production system Feedback ◉ Architecture validation (fitness functions) ◉ Instrumentation / Observability 38
  • 39.
  • 40.
  • 41.
  • 42.
    Nextail 2018-Now ◉ Deploymentfrequency 1-3/day, cloud, SaaS ◉ On our way to XP & DevOps ◉ TDD (not always) ◉ Pairing/Mob (>30%) ◉ Collective ownership (Product/Code) ◉ Trunk base development (not always) ◉ Hexagonal architecture (for new services) ◉ Splitting the monolith 42
  • 43.
    Nextail 2018-Now ◉ ContinuousIntegration: Partially (on the way) ◉ Continuous Delivery: For all the services ◉ Continuous Deployment: Not yet ◉ Infrastructure as a Code: Terraform ◉ Immutable infrastructure: Yes (aws: eks k8s) ◉ Deployment patterns: ○ Canary releasing, Rolling Deploys 43
  • 44.
    TheMotion 2015-2018 ◉ Deploymentfrequency 5-10/day, cloud, SaaS ◉ XP & DevOps practices ◉ TDD ◉ Pairing/Mob (>70%) ◉ Collective ownership (Product/Code) ◉ Trunk base development ◉ Hexagonal architecture ◉ Microservices, Event driven architecture 44
  • 45.
    TheMotion 2015-2018 ◉ ContinuousIntegration: For all the services ◉ Continuous Delivery: For all the services ◉ Continuous Deployment: Not for all services ◉ Infrastructure as a Code: Ansible, Terraform ◉ Immutable infrastructure: Yes (aws: ECS) ◉ Deployment patterns: ○ Canary releasing, Rolling Deploys 45
  • 46.
    Alea Soluciones 2012-2016 ◉Deployment frequency 5-10/week, on-prem ◉ XP & DevOps practices ◉ TDD ◉ Pairing/Mob (>70%) ◉ Collective ownership (Product/Code) ◉ Trunk base development ◉ Hexagonal architecture ◉ Service-oriented architecture 46
  • 47.
    Alea Soluciones 2012-2016 ◉Continuous Integration: For all the services ◉ Continuous Delivery: For all the services ◉ Continuous Deployment: Not for all services ◉ Infrastructure as a Code: Puppet, Ansible ◉ Immutable infrastructure: Partially (docker) ◉ Deployment patterns: ○ Canary releasing, Zero downtime deployment 47
  • 48.
    and now, let’sdiscuss @eferro https://www.eferro.net Thank You! 48

Editor's Notes

  • #5 Technology changes, the world changes... Everything changes all the time and a fast pace. An is accelerating. The only constant is change. The Manifesto for Agile Software Development is from 2001. A lot of things changed since then... Cloud, Web-Scale, Lean startup, Serverless, kubernetes. Microservices, Single pages application, Machine Learning...
  • #6 Sice 2014, there is research about how high performing tech organizations works. The DORA research, conducted by Dr. Nichole Forsgren, and with help from Jez Humble, Gene Kim. The research continues, and each year they present the conclusions about the engineering practices, culture, and other factors that predicts high performance organizations, in the DORA report. Also, in 2016, they published the Accelerate book, explaining the outcomes, the scientific methods used, and some notes about how to introduce some of the engineering practices that the researchers identify as fundamental. The Accelerate book and each one of the annual reports are excellent. This is the best resource we have to understand how high-performance tech organizations work.
  • #7 As they need a way to compare the different companies, they defined a common metrics to measure the Software Delivery and Operational performance. They divide the measures into two groups, throughput, and stability. For throughput, the selected deploy frequency, lead time for changes, and for stability, mean time to restore service, and change failure rate.
  • #8 They discovered that when they measure these KPIs, four groups appear, elite, high performers, medium and low performers. And as we can see the elite are the ones that can deploy on-demand, that lead time from commit to production is less than an hour, they can recover any service in less than an hour and have a low change fail rate. We can see that the low performers are coming from the past… deploying in weeks or months, with serious problems to recover from a failure and despite having less deployment (or perhaps because of it) that fail half of the time they go to production (WTF).
  • #9 We all know that if we increase the frequency of deployment stability suffers and that if we try to be more stable, all we can do is slow down. True... NO! This year is the sixth consecutive year that the study confirms, that high-performance teams can have an excellent throughput and at the same time have high stability.
  • #10  But how... By Introducing Built-in quality. Automated testing, clean code, simple design, etc.
  • #11 Another myth that falls.
  • #13 This diagram is a summary of the capabilities, engineering practices, and processes that the study has found to predict organizational performance. In the technical part, continuous delivery is the fundamental capacity. On the other hand, an agile/lean management style, complemented by a lean product development process and in an environment with a healthy culture, is what predicts that performance.
  • #14 As you can see, basically the engineering practices that predict continuous delivery and predict organizational performance are agile technical practices (updated with DevOps). Of course, these practices involve taking agile engineering and discipline seriously in software development. All of these disciplines introduce quality in the process and shorten the feedback cycle.
  • #15 This slide shows the difference between the elite performers and the low performers. The numbers speak for themselves, and the difference is huge...
  • #16 The good news is that the profession is improving year after year and there are more and more elite teams. So clearly it can be done.
  • #17 In summary… Agile works Devops works We already know the engineering practices. We already know how to develop with quality-in. We already know how to work in short cycles.
  • #21 WTF!
  • #25 Are you agile? Are you doing agile? Are you doing TDD, Pairing, Trunk based development, Hexagonal architecture, loosely coupled architecture, Infrastructure as Code, Immutable infrastructure?
  • #30 Continuous delivery is not cowboy coding or going crazy making random changes. Increasing the deployment frequency without introducing good practices is very risky.
  • #31  It’s a discipline Agile is a discipline It's simple, it's not easy. Like going to the gym
  • #41 Be agile, my friend…