Presentation by Software Engineering Manager and DevOps Coach IBM Marketplace Engineering, Ann Marie Fred. Adopting a new culture and a new way of working isn't easy; if it was, we'd all be working in Shangri-la by now. Adopting a new culture within a company with roughly 400,000 employees is even more difficult. From its humble beginnings with the first two-pizza DevOps team, IBM's DevOps community has grown to thousands of practitioners. I'll talk about balancing interdependencies with independence, and management with freedom. I'll also outline several practical steps you can take to drive change within your own organization, especially when you encounter resistance to change or misguided processes.

1. How IBM Scaled DevOps:
The IBM Marketplace and
Continuous Improvement
Ann Marie Fred
Software Engineering Manager
and DevOps Coach
IBM Marketplace Engineering
Place
Your
Picture
here
February 15, 2016

2. Objectives
By the end of this session, you should be able to:
• Describe characteristics of one exemplary team
• Identify some agile best practices that work well for IBM
• Understand how we safely deploy changes to production many times
per day
• Address challenges with scaling DevOps to a large organization
• Jump-start a DevOps transformation and continuously improve your
own organization
2

3. Agenda
• A bit of history: DevOps early adopters at IBM
• IBM Marketplace Engineering culture and heritage
• Delivering to production, frequently and safely
• How to rebuild an organization
• Getting through roadblocks
3

4. Introduction
Why is DevOps important to IBM?
• To compete with cloud-native companies, we need to streamline our
engineering and operations.
• To foster highly skilled, engaged teams.
• To further a culture of innovation and excellence.
4

5. A bit of history:
DevOps early adopters at IBM
5
2011 2012 2013 2014 2015

6. IBM Marketplace culture and heritage
Characteristics of IBM:
• Hierarchical
• Federated
• Diverse
6

7. IBM Marketplace culture and heritage
Where we are:
• Cloud
•Digital
– Marketplace
– Engineering
7

8. IBM Marketplace
8

9. My IBM
9

10. My Products and Services
10

11. Heritage
11
• Many thanks to…

12. Heritage
• Squad model: inspired by Spotify
•More later…
• Just culture: inspired by Etsy
•Blameless post-mortems
• Microservices: inspired by Netflix
12

13. Heritage
• Lean and Kanban: inspired by Toyota
•Minimum Viable Product (MVP)
•Limit work in progress
•Pull model
13

14. Heritage
• Scrum: long history of success within IBM
•Helps coordinate work across squads
•Weekly sprints
•Rational Team Concert
• Design Thinking: IBM-wide initiative
•Design Thinking: Build the right thing
•DevOps: Build the thing right
14

15. What is the scale?
• IBM Digital (parent) is roughly 500 people, approximately 80 squads
• IBM Marketplace Engineering is roughly 100 people, approximately
20 squads
• DevOps adoption & “Squad-ification” varies, even within Marketplace
Engineering
15

16. Making a huge company feel small
Our version of the squad model:
• Small and focused
• Co-located
• Independent
• Accountable
• Autonomous
• Self-managing
16

17. 17

18. What makes us tick?
Squad Roles
• Executive coach
• Full-stack developers: develop, test, deploy, operate, support
• Tech lead and other committers
• Dedicated designers (visual, ux, industrial)
• Squad lead / product manager / product owner
• Project manager
• HR manager / release manager / engineering discipline
• Scrum master: rotates
18

19. The right tools for the job
Experiment, within limits
• Free tools and frameworks, no hosting cost
• CIO’s office, Whitewater, DevOps Enablement teams
• Purchased SaaS tools
• Tools hosted by other ops teams
• Purchased on-premise tools
• Host-your-own
19
EASY
HARD

20. 20

21. Delivering to production, frequently and safely
21

22. Characteristics of Continuous Delivery
• Tiny, frequent changes
• Carefully tested and reviewed
• Delivered to production, automatically
• During normal business hours
• Zero downtime
22

23. Continuous Delivery reduces risk
• Quick fixes
• Quick feedback
• Gradual improvement
23

24. Continuous Delivery without fear
These things work together:
• Architecture
• Development and Test practices
• Deployments
• Operations
24

25. Continuous Delivery: Architecture
• Resilient microservices
•Small, easy to maintain
•Multiple instances, multiple locations
•Fail-over
•Graceful degradation
• Bluemix PaaS
•We handle the application/service, they
handle the infrastructure/operating
system
25

26. Continuous Delivery: Development and Test
• Stringent code reviews
• 100% unit test code coverage
• Automated integration tests
• Automated system tests in production
• Automated security tests: Rational AppScan
26

27. Continuous Delivery: Deployments
• Automated deployments
• Blue/green deployments
27

28. Continuous Delivery: Operations
• Streamlined release management
•Compliance
•HA/DR plans
•Operations and support plans
•At launch, then update periodically
• Monitoring
• Alerting
• Logging
• Feedback (see next slide)
28

29. Continuous Delivery: Feedback
• A/B Testing
• Monitoring feedback
29

30. How to build a DevOps organization
This is awesome, but…
• Where did we start?
• How did we scale up?
30

31. Pioneer teams
31

32. Pioneer teams
• Start from scratch
• Strong leaders
• Learn, try, adapt, reject, repeat
• No cynicism or defeatism
• Do the right thing, ask forgiveness later
32

33. Viral effect
• From one squad, one project, to many
• Simplify access to best tools
• Leaders and SMEs teach others
• External community support
• Formal training
• Cross pollination
• Adopt and adapt
33

34. Learning from failures
“Anyone who has never made a
mistake has never tried anything
new.” – Albert Einstein
Continuous improvement:
• Blameless post-mortems
• Squad health surveys
• Engineering Summit
• Retrospectives
34

35. Getting through roadblocks
Case studies
•Too much red tape
•This is taking too long!
•I want more plans!
•We’re fine, thanks.
•Two-speed IT
35

36. Questions?
36

37. Review of Objectives
Now you can:
• Describe characteristics of one exemplary team
• Identify some agile best practices that work well for IBM
• Understand how we safely deploy changes to production many
times per day
• Address challenges with scaling DevOps to a large organization
• Jump-start a DevOps transformation and continuously improve
your own organization
37

38. Contact Info
• amfred@us.ibm.com
• @DukeAMO
Thank You!
38