Author: Adrian Perreau de Pinninck
Conference: Global Scrum Gathering Berlin 2014
Describes how Criteo managed to change the software engineering culture of its R&D department from 'Testing is cheating' to 22K+ tests being run every half hour.
2. TREMENDOUS GROWTH.
CRITEO: GO GO GO!!!
• From 20 to 160 engineers in less than 5 years
• 160 code repositories
• 7 million lines of code
• Took months to release
3. TO PUT THE MACHINES UNDER CONTROL
WE MUST SEE REALITY
4. WHAT WAS REALITY?
• Average lead time for new features = 60 days
• Average time to merge commits = 20 days
• The release pipeline was blocked for months
• It was getting worse each year
5. IT WAS GOING TO TAKE TOP MANAGEMENT TO AGREE ON SOMETHING LIKE THIS
SOMETHING HAD TO CHANGE
7. OBJECTIVES
• Reduce commit merge time to hours instead of days
• Reduce time from commit to Prod to < 4 days
• Reduce number of cancelled release candidates
8. TECHNICAL SOLUTION
• Tailor-made Build System
• Mandatory Code Reviews
• Continuous Integration Pipeline
• Sandbox to execute tests
10. ESTABLISH SENSE OF URGENCY
• Code freeze in 2012 crippled the team during months
• The release pipeline had been blocked for months
• Things weren’t getting better for 2013’s code freeze
12. STRONG GUIDING COALITION
• Executive Vice President of Engineering
• Vice President of QA
• 10 Software Engineers, high level of seniority
• 2 Senior Engineering Program Managers
13. DEVELOPING A CHANGE VISION
• Brainstormed for the key elements
• Prepared a document explaining the vision
• Shared the document with Key players
• Modified the document to take into account feedback
• Shared the document with Dev Leads
• Modified the document to take into account feedback
• Gave presentation explaining the project
• Sent document to all
14. CHANGE VISION
• 26 page document
• Took over a month to get
alignment in version 0.1
• Sent to 20 senior
engineers for feedback
• Last version took into
account all feedback
15. COMMUNICAT E
THE VISION
• Monthly updates at all
hands meetings
• Weekly emails with
progress reports
• Events with food
17. EMPOWER
ACTION
• The improvement backlog
was shared with everyone
• Some items prioritised by
voting
• Voluntary-based work
encouraged
• Test-a-thons and other
wide range initiatives
18. SHORT-TERM
WINS
• Used Scrum to guide the
change process
• Developed usable tools from
the beginning
• Developed one tool at a time
• Pushed adoption slowly until
tipping point
• Forced rest to adopt once
tipping point passed
22. 100"
90"
80"
70"
60"
50"
40"
30"
20"
10"
0"
FROM 35 DAYS TO 15 DAYS
COMMIT AGE
Commit&to&Prod&Lag&
5/13/13" 6/13/13" 7/13/13" 8/13/13" 9/13/13" 10/13/13" 11/13/13" 12/13/13" 1/13/14" 2/13/14" 3/13/14" 4/13/14" 5/13/14"
23. I T ’ S A LWAYS HARDER THAN YOU THOUGHT
DON’T LET GO
24. LARGEST IMPEDIMENTS
• Technical
• Eat your own dog food
• Replicating the Sandbox
• Human
• People added to the team not sharing vision
• Getting teams to use the tools
• Managing expectations and communication
25. DON’T LET GO
• Fully dedicated team for a whole year.
• After the project ended the team remained as a new
unit.
• Even through the hardest times management gave full
support.
27. INCORPORATE CHANGE
INTO THE CULTURE
• Training program for new recruits
• Engineering partners are expected to conform to new
way of working
• Training program for engineering partners
• Tools in place become the cultural gates
29. WE ASKED THOSE ALREADY ONBOARD TO HELP OTHERS DO THE SAME
BRIDGE BUILDER
30. CHAMPION
SKEPTIC
Find someone senior in the
organisation:
• That wants the goal to be
reached
• That doesn’t fully support
your technical solution
• Talk with him regularly