4. A better TTM (Time To Market)
!
Faster from business idea to production
!
Faster to evolve the systems, the technology
!
… while keeping a HIGH level of quality, stability, availability, operability
4
6. Local goals are divergent
DevOps
«
wall
of
confusion
»
Local
goals
Local
culture
Deliver
new
features
(of
good
quality)
Guarantee
the
run
of
so;ware
(stability)
Product
oriented
Service
oriented
(supervision,
backups,
provisioning…)
Try
to
innovate
Try
to
raBonalize
6
7. Activities, on Ops side
§ 47%
of
total
Bme
is
dedicated
to
deployment
§ Doing
the
deployment
§ Fixing
problems
related
to
deployments
§ InteresBng
KPI
to
follow…
7
9. !
!
!
800 dev, 400 ops, 180’000 servers (= 450 servers / ops)
1 deployment each day
Concept of deployment rings
!
!
A deployment somewhere in datacenters every 11 seconds
At any moment, an average of 10’000 servers are being
updated
!
!
Everything is in the cloud (AWS)
« Design For Failure » :
! the software is designed to tolerate
! they test it all the time, in production.
! 65’000 failure tests, in production, by killing random virtual
machine !!! … and measuring that everything is alright !
!
10 deployments / day
9
13. !
Because humans make mistakes
!
Because human brain is terribly bad at repetitive tasks
!
Because human is slow compared to a bash script
!
… and because we are humans
13
15. Command and
Control
Configuration
Application Orchestration
• deploy applicative code
• … and rollback !
• deploy database scripts and data
• start application
• join clusters…
System Configuration
• jvm, application servers…
• middlewares…
• service configuration (logs, ports,
user and group permissions)
• registration to supervision
Capistrano
Custom shell scripts
...
Chef
Puppet
CFEngine
…
OpenStack
Bootstrapping
•
•
VM Instantiation
OS setup & configuration
Vmware vCloud
Oracle exaLogic
or physical machine config
…
15
16. Command and
Control
Configuration
Application Orchestration
• deploy applicative code
• … and rollback !
• deploy database scripts and data
• start application
• join clusters…
System Configuration
• jvm, application servers…
• middlewares…
• service configuration (logs, ports,
user and group permissions)
• registration to supervision
Capistrano
Custom shell scripts
...
Chef
Puppet
CFEngine
…
OpenStack
Bootstrapping
•
•
VM Instantiation
OS setup & configuration
Vmware vCloud
Oracle exaLogic
or physical machine config
…
16
17. Command and
Control
Configuration
Application Orchestration
• deploy applicative code
• … and rollback !
• deploy database scripts and data
• start application
• join clusters…
System Configuration
• jvm, application servers…
• middlewares…
• service configuration (logs, ports,
user and group permissions)
• registration to supervision
Capistrano
Custom shell scripts
...
Chef
Puppet
CFEngine
…
OpenStack
Bootstrapping
•
•
VM Instantiation
OS setup & configuration
Vmware vCloud
Oracle exaLogic
or physical machine config
…
17
18. Capistrano
!
Consider all of this like code
Custom shell scripts
...
!
This can be stored and versioned in a code
repository (like git)
Chef
!
!
Like software code, you can test it. And make
sure your script won’t fail when you need it at
2:00 am
Can scale to any number of server in parallel…
Puppet
CFEngine
…
OpenStack
Vmware vCloud
Oracle exaLogic
or physical machine config
…
18
19. Infrastructure as Code
!
Benefits of « Infrastructure as Code »
! REPETABILITY and RELIABILITY
!
Guarantee that infrastructure is homogeneous
!
Make sure that standards are respected
!
Any new environment is configured quicker
!
Allow developers to do lots of tasks by themselves
!
eg. in qualification environment
19
21. !
!
!
800 dev, 400 ops, 180’000 servers (= 450 servers / ops)
1 deployment each day
Concept of deployment rings
!
!
A deployment somewhere in datacenters every 11 seconds
At any moment, an average of 10’000 servers are being
updated
!
!
Everything is in the cloud (AWS)
« Design For Failure » :
! the software is designed to tolerate
! they test it all the time, in production.
! 65’000 failure tests, in production, by killing random virtual
machine !!! … and measuring that everything is alright !
!
10 deployments / day
21
27. What you need :
! Continuous integration
! TDD - Test Driven Development
(automated unit testing)
! Code reviews
! Continuous code auditing (sonar…)
! Functional test automation
! Strong non-functional tests
(performance, availability…)
! Automated packaging and deployment,
independent of target environment
27
28. You may also need
! Feature flipping / Feature toggles
Feature flipping allows to enable / disable a
feature while software is running
28
29. You may also need
Dark Launch @ Facebook :
! Feature flipping / Feature toggles
Dark launching is releasing a new feature, with no
UI changes, but using all the parts of your
infrastructure involved in serving that feature.
! Dark Launch
A good strategy to apply when you're dealing with
massive, large-scale deployments, and when you
want to see how your infrastructure behaves in
conditions that are as close to production as
possible
29
30. Canary Release
You may also need
! Feature flipping / Feature toggles
Users
! Dark Launch
Router
! Canary Release
Most of users
Version N
Subset of users
Version N+1
30
31. Blue/Green deployment
You may also need
! Feature flipping / Feature toggles
! Dark Launch
Web server
App server
DB server
! Canary Release
! Blue/Green deployment
Router
Users
31
32. You may also need
Datamodel evolution strategy example
! Feature flipping / Feature toggles
! Dark Launch
Deploy application
compatible with
N and N+1
! Canary Release
! Blue/Green deployment
! Datamodel evolution strategies
DB
Prepare hybrid
datamodel :
version N
with N+1 things
CREATE TABLE
ADD COLUMN
DB
Remove old things
Datamodel =
version N+1
DROP TABLE
DROP COLUMN
32
33. Datamodel evolution strategy example
Datamodel
Version N
Datamodel
Version N
V.1
Datamodel
Version N+1
Hybrid
V.1 + V.2
Datamodel
Version N+1
V.2
33
37. Ops are the others users of the system
Business
Tester
End-‐user
funcBonal
tests
(automated)
FuncBonal
requirements
PriorizaBon
Development
So;ware
Ops
37
38. Ops must be involved in all the phases of a project
Business
Tester
End-‐user
funcBonal
tests
(automated)
-‐
FuncBonal
requirements
-‐
Non
funcBonal
requirements
(useful
logs,
degraded
modes,
redo
operaBons,
integraBon
to
supervision,
performance,
easiness
of
deployment)
So;ware
PriorizaBon
Development
non-‐funcBonal
tests
(automated)
Ops
Manager
Ops
Ops
38
39. Share the tools
Ops
Dev
-‐ Binary
repositories
-‐ ConBnuous
integraBon
-‐ Facilitate
/
Automate
deployment
-‐ Help
dev
teams
to
become
autonomous
(in
parBcular
for
deployment
before
producBon)
-‐ Facilitate
diagnosBcs
-‐ Get
devs
interested
in
«
prod
things
»
-‐ Give
access
to
logs
-‐ Give
access
to
supervision
and
monitoring
39
40. Des outils partagés, qui facilitent les interactions
Get
devs
interested
in
«
prod
things
»
3.
COLLABORATIO
N
(culture,
organisaBon…)
40
41. Work together
Architecture
reviews
align
people
on
objecBves
Post
Mortem
idenBfy
root
causes,
without
blaming
people
improve
the
way
you
work
Day-‐to-‐day
share
informaBon
ReporBng
Go
/
NoGo
communicate
about
how
the
system
works
anBcipate
problems
decision
to
go
in
producBon
should
be
a
shared
decision
(markeBng,
dev,
ops,…)
41
43. tools
for
collaboraBon,
measurement,
quality
management,
automaBon
architectures
adapted
to
funcBonal
AND
non-‐funcBonal
requirements
a
set
of
pracBces
that
aim
at
improving
collaboraBon
between
dev
and
ops
culture
and
organizaBon
to
ease
collaboraBon
methodology
for
collaboraBon,
trust,
and
conBnuous
improvement
43
47. Progressive implementation
Improve
Quality of Service
Continuous
Improvement
Culture of
collaboration
Speed-up
the provisioning
Infrastructure
as Code
Improve
deployment
reliability
We
advocate
for
a
progressive
and
iteraBve
implementaBon,
adopBng
the
principles
of
conBnuous
improvement
from
the
lean
management
Continuous
Delivery
Improve
Time-To-Market
Improve
Mean Time
To Recover
(MTTR)
47
48. Diagnostic : Where are you ?
Example of an evaluation
matrix we use : tooling,
processes, architecture,
organization, overall
performance…
48
49. OCTO Services Offers
Directors
Talks & Seminars
• “Evangelisation”
Diagnostics & Evaluation
• DevOps Maturity
• Organizational Audit
• Practices Audit
Managers
Architects
(so;
&
infra)
Devs
Ops
Architecture studies
• DevOps patterns (Dark Launch...)
• Software Delivery pipeline
• Deployment architecture
Coaching
• State of the art practices (XP, TDD, test for ops)
• Agile/Kanban for ops
• Animation/Facilitation
Expertise : Technical Assistance & Training
• Continuous Delivery platform setup
• Infrastructure as Code (vagrant, chef, cap...)
49
