Moving 65,000 Microsofties to DevOps with VSTS
Willy-Peter Schaub

Willy[-Peter] Schaub
Program Manager | Developer Ops Canada
willys@microsoft.com
@wpschaub

I work
here
I was born here
I grew up here

Truk Lagoon
Semiahmoo Bay
Protea Banks
Hawaii
Wondergat
Aliwal Shoal
Truk Lagoon

DevOps
Visual Studio
Application Insights
Any language, Any Platform

One Engineering System using VSTS

What is DevOps?
Increase flow of value Shorten cycle times Continuously Improve

Our Definition of Done
Customer Focused

STARTOF OUR JOURNEY
August 2010
Sprint 1
Extensions
Nov 2015
Sprint 931ES
June 2014
Sprint 67
GVFS
June 2016
Sprint 102
Ranger community
adopted VSTS
April 2011
VSTS
April 2014
Sprint 64
over night 
~destination! 
continuous innovation! 

Five habits we’ve learned so far
++
++
++

Five habits we’ve learned so far

Listen to our customers
Customer Focused

Collect data
broadly (but carefully)

But measure what’s important (KPI’s)
Customer Focused
• Original estimate
• Completed hours
• Lines of Code
• Team capacity
• Team burndown
• Team velocity
• # of bugs found
Things we don’t watch
• Acquisition
• Engagement
• Satisfaction
• Churn
• Feature Usage
Usage
• Time to Detect
• Time to Communicate
• Time to Mitigate
• Customer Impact
• Incident Prevention Items
• Aging Live Site Problems
• SLA per Customer
• Customer Support Metrics
Live Site Health
• Time to Build
• Time to Self Test
• Time to Deploy
• Time to Learn
Process Velocity

Maximize learning and influence value
Customer Focused
Validatedlearning
Deployment frequency
PROVEN DISPROVEN
VALIDATE
business
service
value

Feature Flags – fine tune user experience
Customer Focused
if ( flag )
else

Five habits we’ve learned so far

Live Site Incidents

Be Transparent
Production First Mindset

Alerting is the key to fast detection
Production First Mindset
Before
• Redundant alerts for same the issue
• Needed to set right thresholds and
tune often
• Stateless alerts contributed to
further noise
After
• Every alert must be actionable and
represent a real issue with the
system.
• Alerts should create a sense of
urgency – false alerts dilutes that
• Use alerts to auto-dial the DRIs.

Automate completely

Tracking Deployments to Production

Circuit Breakers
https://github.com/Netflix/Hystrix/wiki

Security Mindset
Production First Mindset
 Double blind test
 Full disclosure at or near end
vs.
 Share tactics & lessons learned
 Continued evolution
Assume Breach - Use War Games to the learn attacks and practice response

Five habits we’ve learned so far

Agile at Scale with Aligned Autonomy
Team Autonomy & Enterprise Alignment
Alignment
within the
business
Team
autonomy
“Let’s try to give our teams three things….
Autonomy, Mastery, Purpose”
Anarchy

Team Autonomy & Enterprise Alignment

Roles
Team Autonomy & Enterprise Alignment
Program Management Dev Test

Roles
Team Autonomy & Enterprise Alignment
Program Management Engineering

Yes, there are other roles…
Team Autonomy & Enterprise Alignment
Program Management Engineering Service DeliveryUX UE
Service Delivery is integrated
directly into our organization.

Teams
Team Autonomy & Enterprise Alignment
Program Management is responsible for:
WHAT we’re building, and
WHY we’re building it
Engineering is responsible for
HOW we’re building it, and that
we’re building it with QUALITY

Teams
 Physical team rooms
 Cross discipline
 10-12 people
 Self managing
 Clear charter and goals
 Intact for 12-18 months
 Own features in production
 Own deployment of features

Instead of Horizontal…
Team Autonomy & Enterprise Alignment
UI
API
Data

We strive for Vertical
Team Autonomy & Enterprise Alignment
UI
API
Data

Employee choice, not
manager driven
Typically <20%
change, but 100% get
to make a choice
Cross-pollinate talent
and micro-culture
Sticky Note Exercise - Self Forming Teams

Sprint
3 weeks
confident
1
Plan
3 sprints
thoughtful
3
Season
6 months
hopeful
6
Planning
Team Autonomy & Enterprise Alignment
Scenario
18 months
asprirational
Teams are responsible for the detail
Leadership is responsible
for driving the big picture

Planning
Team Autonomy & Enterprise Alignment
Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4
18 month scenario
6 month plan

Planning
Team Autonomy & Enterprise Alignment
Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4
18 month scenario
6 month plan

Planning
Team Autonomy & Enterprise Alignment
Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4
18 month scenario
6 month plan

Planning
Team Autonomy & Enterprise Alignment
Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4
18 month scenario
6 month plan

Traditional Planning
Team Autonomy & Enterprise Alignment
TIME
VALUE

Plan, Deliver, and Learn
Team Autonomy & Enterprise Alignment
TIME
VALUE

Let’s Compare
Team Autonomy & Enterprise Alignment

What we accomplished
3 week sprints
Team Autonomy & Enterprise Aligmen
Progressive
Deployment
Week 1 Week 2 Week 3
Week 1 Week 2 Week 3Week 2 Week 3
Sprint 98Sprint 97 Sprint 99
The sprint plan
930!

Sprint mails

Team Chats
Team Autonomy & Enterprise Alignment
SpringFallSpring Fall
3 sprints

Direct. No “lost in translation”.
Team Autonomy & Enterprise Alignment

Experience Reviews
Team Autonomy & Enterprise Alignment
SpringFallSpring Fall

Benefits
Team Autonomy & Enterprise Alignment

Five habits we’ve learned so far

engineers on
your team#
Bug Cap
Shift Left Quality
5 ?x =

Bug Cap
Shift Left Quality
Rule: If your bug count exceeds your bug cap… stop working
on new features until you’re back under the cap.
5 50x =10

Code Test & Stabilize Code Test & Stabilize
Code
Complete
Planning
Before – Debt Cycle
Shift Left Quality

After
Shift Left Quality

• Tests that anyone can run anywhere (inc production)
• Shifted to unit tests from automated functional tests
• Core tests run before pull request
• Fast and 100% reliable build and test is critical
• Rolling tests run after commit
Test Portfolio - Shift Left from Integration to Unit

Pull Requests 

Release Flow Branching Structure
Shift Left Quality
LSI

Five habits we’ve learned so far
Customer Focused
Production First Mindset
Team Autonomy + Enterprise Alignment
Shift Left Quality
Infrastructure as Flexible Resource

From Labs to VM’s to DevTest Labs

Multiple Data Centers with incremental roll out
Shared Platform Services (SPS)
North Central
TFS SU1
North Central
AT
AT
AT
JA
JA
JA
Blob
TFS SU7
Australia
TFS SU0
West Central
Containerized Services

Architecture Modernization
•
•
•
•
•
• If you are starting out today and cloud native, consider
PaaS, Service Fabric + Azure Functions
• We need to ship same code to on-prem & cloud

DevOps isn’t magic

Our DevOps Transformation – the story so far
After
• 3-week sprints
• Vertical teams
• Team rooms
• Continual Planning & Learning
• PM & Engineering
• Continual customer engagement
• Everyone in master
• 8-12 person teams
• Publicly shared roadmap
• Zero debt
• Mockups in PPT
• Inner source
• Flattened organization hierarchy
• User satisfaction determines success
• Features shipped every sprint
Before
• 4-6 month milestones
• Horizontal teams
• Personal offices
• Long planning cycles
• PM, Dev, Test
• Yearly customer engagement
• Feature branches
• 20+ person teams
• Secret roadmap
• Bug debt
• 100 page spec documents
• Private repositories
• Deep organizational hierarchy
• Success is a measure of install numbers
• Features shipped once a year

Lastly the DevOps Ranger Transformation
DevOps even works for a part-time open source community 
After
• 2-5 person teams
• 0.25 program managers
• Automated CI
• Automated CD
• 3-5 sprints cadence
• 3-week sprints
• Proactive telemetry
• Minutes to days to resolve issues
• Minutes to build
• Minutes to release
Before
• 10-15 person teams
• 2 program managers
• Manual and error prone builds
• Manual and error probe releases
• 6-12 sprint cadence
• 1 month sprints
• Issues detected by users
• Days to weeks to resolve issues
• Hours to build
• Days to release

Quick Reference Posters – ping me if you’re interested in a digital copy!

Real-world demo





 

/* THANK YOU*/
WILLY SCHAUB
willys@microsoft.com
@wpschaub
aka.ms/devops
aka.ms/devopsassessment
aka.ms/vsarpublications
@VSTS
http://www.devconf.co.za/