DevOps is a Reorg How To Accelerate The Change

1. © Copyright 2015 EMC Corporation. All rights reserved. 1© Copyright 2015 EMC Corporation. All rights reserved.

2. © Copyright 2015 EMC Corporation. All rights reserved.
DevOps is a Reorg
How To Accelerate The Change
Don Demcsak
East Region – DevOps Process Lead
Twitter - @DonDemcsak

3. © Copyright 2015 EMC Corporation. All rights reserved.
CHANGE IS THE NEW BUSINESS NORM
3© Copyright 2015 EMC Corporation. All rights reserved.

4. © Copyright 2015 EMC Corporation. All rights reserved.
AGILITY – BAKING CHANGE INTO IT
Organization Design
Org
Agility
Business Agility
People Process Tools
IT
Agility

5. © Copyright 2015 EMC Corporation. All rights reserved. 5© Copyright 2015 EMC Corporation. All rights reserved.
STRATEGIES FOR MANAGING CHANGE

6. © Copyright 2015 EMC Corporation. All rights reserved.
• From Silos to Delivery Teams
– Outcomes over Activities
• This change isn’t “solved” with a
technology change
• This change is evolved through:
People + Process + Technology
Culture!
THE FOCUS OF IT NEEDS TO CHANGE

7. © Copyright 2015 EMC Corporation. All rights reserved.
PEOPLE + PROCESS + TECHNOLOGIES
Aligning ALM and Cloud strategies

8. © Copyright 2015 EMC Corporation. All rights reserved.
UNDERSTANDING CHANGES IN IT
• Fictional story about today’s IT challenges
• Explains how Operational and Cultural changes save
the day.
• Based on “The Goal: A Process of Ongoing
Improvement” book
• Mark Burgess, a pioneer of system configuration
management. Creator of CFEngine.
• What Science tell us about Information Infrastructure
video (Video)

9. © Copyright 2015 EMC Corporation. All rights reserved.
BI-MODAL or TRI-MODAL IT?
TRI-MODAL (SIMON WARDLEY)BI-MODAL (GARTNER)
BUSINESS
IT MANAGEMENT
LEGACY
APPLICATIONS
NEW
APPS

10. © Copyright 2015 EMC Corporation. All rights reserved.
• Eliminate non-value-added action (Waste/Muda)
• Pull over Push
• Kaizen (Continuous Improvement)
• Kaikaku (Disruptive Change)
• Small Batch + Experimentation
• Kanban (Try www.trello.com )
• Iterative vs. Incremental
ADOPTION OF LEAN PRACTICES
http://en.wikipedia.org/wiki/Lean_startup

11. © Copyright 2015 EMC Corporation. All rights reserved.
BUILD PROJECTS ITERATIVE VS. INCREMENTAL
Painting of
the Mona Lisa
Woman in
Pastoral Settings
1 2 3
Incremental
Iterative

12. © Copyright 2015 EMC Corporation. All rights reserved.
the “WHAT” – defines outcomes; sets
enterprises standards to maintain quality
and provide a consistent work environment
across the enterprise
EMPLOY A TOP DOWN & BOTTOMS UP CHANGE MODEL
SUCCESS REQUIRES A SHARED VISION
CHANGE ADOPTION
LEADERSHIP TEAM
DELIVERY TEAM
the “HOW” – determines the best
approach for achieving the standards
defined at the enterprise level

13. © Copyright 2015 EMC Corporation. All rights reserved.
• Reduce cycle time by 300%
• Reduce recovery/rollback time to less than 4hrs
• Improve process efficiency by 10%
• Reduce break/fix workload from 50% to 30%
• Solution is extensible and parameterized
SUCCESS CRITERIA & MEASURES
DEFINE THE OUTCOMES OF TRANSFORMATION

14. © Copyright 2015 EMC Corporation. All rights reserved.
Release Candidate Pipelines
DELIVERY PIPELINES
VERSION
CONTROL
COMMIT
CREATE
CHANGE
• New software
• Enhancements/fixes
• Upgrades
• Patches
• Configurations
TEST UAT STAGE
PROD
Development & Build Process
1.1
1.2
1.3
1.4
1.5
1.2
Automated Tool Chain

15. © Copyright 2015 EMC Corporation. All rights reserved.
START SMALL AND SCALE WITH CONFIDENCE
BUILDING PIPELINES - INCREMENTAL METHODOLOGY
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APPAPP
APP
APP
APP
APP APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
APP
Single Application
(Project Team)
Dependent Applications
(Program Teams)
10-100s Applications
(Line-of-Business Portfolio)
100-1000s Applications
(Enterprise)
PHASE1
PHASE2PHASE4
PHASE3
GOAL
GOAL
GOAL
GOAL
Goal
+ +
+ +

16. © Copyright 2015 EMC Corporation. All rights reserved.
OPTIMIZING CHANGE MANAGEMENT
Release
Frequency
1 Release Every 100
Days
1 Release Every
10 Days
Daily Release 10 Releases a
Day
100 Releases a
Day
Infrastructure
Production-like environments can be prepared
thru version controlled scripts
Heterogeneous, managed
by operations
Production-Standardized PaaS or IaaS provided by delivery
team

17. © Copyright 2015 EMC Corporation. All rights reserved.
OPTIMIZING CHANGE MANAGEMENT
Release
Frequency
1 Release Every 100
Days
1 Release Every
10 Days
Daily Release 10 Releases a
Day
100 Releases a
Day
Infrastructure
App
Architecture
Production-like environments can be prepared
thru version controlled scripts
Heterogeneous, managed
by operations
Production-Standardized PaaS or IaaS provided by delivery
team
Everything deployed
together
Single packet per product/web property
Strict SOA with forward/backward compatibility

18. © Copyright 2015 EMC Corporation. All rights reserved.
OPTIMIZING CHANGE MANAGEMENT
Release
Frequency
1 Release Every 100
Days
1 Release Every
10 Days
Daily Release 10 Releases a
Day
100 Releases a
Day
Infrastructure
App
Architecture
Testing
Production-like environments can be prepared
thru version controlled scripts
Heterogeneous, managed
by operations
Production-Standardized PaaS or IaaS provided by delivery
team
Everything deployed
together
Single packet per product/web property
Strict SOA with forward/backward compatibility
Separate QA department that
owns functional test automation
Fast, comprehensive automated tests at unit and functional level
organized in a deployment pipeline
Developers & Testers maintain automated
functional test together

19. © Copyright 2015 EMC Corporation. All rights reserved.
OPTIMIZING CHANGE MANAGEMENT
Release
Frequency
1 Release Every 100
Days
1 Release Every
10 Days
Daily Release 10 Releases a
Day
100 Releases a
Day
Infrastructure
App
Architecture
Testing
Databases
Production-like environments can be prepared
thru version controlled scripts
Heterogeneous, managed
by operations
Production-Standardized PaaS or IaaS provided by delivery
team
Everything deployed
together
Single packet per product/web property
Strict SOA with forward/backward compatibility
Separate QA department that
owns functional test automation
Fast, comprehensive automated tests at unit and functional level
organized in a deployment pipeline
Developers & Testers maintain automated
functional test together
Manual
migrations
Delta scripts for db,
rollbacks practiced
Apps are forward/backward compatible with db
versions by design (using expand/contract)

20. © Copyright 2015 EMC Corporation. All rights reserved.
OPTIMIZING CHANGE MANAGEMENT
Release
Frequency
1 Release Every 100
Days
1 Release Every
10 Days
Daily Release 10 Releases a
Day
100 Releases a
Day
Infrastructure
App
Architecture
Testing
Databases
Releases
Branch Model
Production-like environments can be prepared
thru version controlled scripts
Heterogeneous, managed
by operations
Production-Standardized PaaS or IaaS provided by delivery
team
Everything deployed
together
Single packet per product/web property
Strict SOA with forward/backward compatibility
Separate QA department that
owns functional test automation
Fast, comprehensive automated tests at unit and functional level
organized in a deployment pipeline
Developers & Testers maintain automated
functional test together
Manual
migrations
Delta scripts for db,
rollbacks practiced
Apps are forward/backward compatible with db
versions by design (using expand/contract)
Release trains
Dark launching, Blue-green deployment, canary releasing
Developers push their own
changes
Develop on branches
Merge to release branches
Then branch again after release
Develop on trunk
Branch for Release
Develop on trunk
Release from trunk
Pull Requests to
a Release Branch

21. © Copyright 2015 EMC Corporation. All rights reserved.
OPTIMIZING CHANGE MANAGEMENT
Release
Frequency
1 Release Every 100
Days
1 Release Every
10 Days
Daily Release 10 Releases a
Day
100 Releases a
Day
Infrastructure
App
Architecture
Testing
Databases
Releases
Branch Model
Production-like environments can be prepared
thru version controlled scripts
Heterogeneous, managed
by operations
Production-Standardized PaaS or IaaS provided by delivery
team
Everything deployed
together
Single packet per product/web property
Strict SOA with forward/backward compatibility
Separate QA department that
owns functional test automation
Fast, comprehensive automated tests at unit and functional level
organized in a deployment pipeline
Developers & Testers maintain automated
functional test together
Manual
migrations
Delta scripts for db,
rollbacks practiced
Apps are forward/backward compatible with db
versions by design (using expand/contract)
Release trains
Dark launching, Blue-green deployment, canary releasing
Developers push their own
changes
Develop on branches
Merge to release branches
Then branch again after release
Develop on trunk
Branch for Release
Develop on trunk
Release from trunk
Pull Requests to
a Release Branch

22. © Copyright 2015 EMC Corporation. All rights reserved.
WHAT DO WE DEFINE AS
INFRASTRUCTURE?

23. © Copyright 2015 EMC Corporation. All rights reserved.

24. © Copyright 2015 EMC Corporation. All rights reserved.

25. © Copyright 2015 EMC Corporation. All rights reserved.

26. © Copyright 2015 EMC Corporation. All rights reserved.
• Deploy, monitor and connect together all the pieces
needed to run services for the organization
• Processes and tools for faster end-to-end delivery of
quality services
• Automation comes built-in
WHY DEFINE INFRASTRUCTURE AS CODE?

27. © Copyright 2015 EMC Corporation. All rights reserved.
• Desired state specified in text files
• Autonomic (self-corrects to desired state)
• State should be known through monitoring
• Remove snowflake servers
HOW DO WE DO THIS?

28. © Copyright 2015 EMC Corporation. All rights reserved.
• Easy to read and edit
• Shareable
• Can use standard version control like Git or SVN
• Becomes executable documentation
WHY STORE THEM IN TEXT FILES?

29. © Copyright 2015 EMC Corporation. All rights reserved.
node 'www2' {
class { 'apache': } # use apache module
apache::vhost { 'awesomewebsite.com': # define vhost
port => '80',
docroot => '/var/www/html'
}
}
SIMPLE EXAMPLE

30. © Copyright 2015 EMC Corporation. All rights reserved.
user { root:
ensure => present,
password => '$ecretP@ssw0rd',
}
CHANGE ROOT PASSWORD

31. © Copyright 2015 EMC Corporation. All rights reserved.
MANUALLY CONFIGURED
ENVIRONMENTS ARE LIKE A HOUSE
OF CARDS IN A CHINA SHOP
- NEAL FORD

32. © Copyright 2015 EMC Corporation. All rights reserved.
WHAT CAN HAPPEN IF YOU
DON'T TREAT
INFRASTRUCTURE AS CODE?

33. © Copyright 2015 EMC Corporation. All rights reserved.
UP TO 60% OF FAILURES ARE
CAUSED BY HUMAN ERROR, NOT
HARDWARE FAILURE

34. © Copyright 2015 EMC Corporation. All rights reserved.
EXAMPLE - KNIGHT CAPITAL GROUP
https://infocus.emc.com/wp-content/uploads/2012/08/KCG.jpg

35. © Copyright 2015 EMC Corporation. All rights reserved.
• Manual deployment of new trading software
• 7 of 8 servers correctly updated
• Old function still alive on the 8th server led to…
KNIGHT CAPITAL GROUP - WHAT HAPPENED?

36. © Copyright 2015 EMC Corporation. All rights reserved.
$440 MILLION LOSS
IN 45 MINUTES

37. © Copyright 2015 EMC Corporation. All rights reserved.
SO

38. © Copyright 2015 EMC Corporation. All rights reserved.
TREAT YOUR
INFRASTRUCTURE AS
CODE