Author: Adrian Perreau de Pinninck Conference: Global Scrum Gathering Berlin 2014 Describes how Criteo managed to change the software engineering culture of its R&D department from 'Testing is cheating' to 22K+ tests being run every half hour.

1. DR. ADRIAN PERREAU DE PINNINCK
TAKE THE RED PI L L

2. TREMENDOUS GROWTH.
CRITEO: GO GO GO!!!
• From 20 to 160 engineers in less than 5 years
• 160 code repositories
• 7 million lines of code
• Took months to release

3. TO PUT THE MACHINES UNDER CONTROL
WE MUST SEE REALITY

4. WHAT WAS REALITY?
• Average lead time for new features = 60 days
• Average time to merge commits = 20 days
• The release pipeline was blocked for months
• It was getting worse each year

5. IT WAS GOING TO TAKE TOP MANAGEMENT TO AGREE ON SOMETHING LIKE THIS
SOMETHING HAD TO CHANGE

6. PAIR DISCUSSION:
HOW WOULD YOU SELL
T H I S I N I T I ATIVE?

7. OBJECTIVES
• Reduce commit merge time to hours instead of days
• Reduce time from commit to Prod to < 4 days
• Reduce number of cancelled release candidates

8. TECHNICAL SOLUTION
• Tailor-made Build System
• Mandatory Code Reviews
• Continuous Integration Pipeline
• Sandbox to execute tests

9. PROCESS FOLLOWED

10. ESTABLISH SENSE OF URGENCY
• Code freeze in 2012 crippled the team during months
• The release pipeline had been blocked for months
• Things weren’t getting better for 2013’s code freeze

11. WITHOUT A STRONG GUIDING COALITION
ENERGY WILL DISSIPATE

12. STRONG GUIDING COALITION
• Executive Vice President of Engineering
• Vice President of QA
• 10 Software Engineers, high level of seniority
• 2 Senior Engineering Program Managers

13. DEVELOPING A CHANGE VISION
• Brainstormed for the key elements
• Prepared a document explaining the vision
• Shared the document with Key players
• Modified the document to take into account feedback
• Shared the document with Dev Leads
• Modified the document to take into account feedback
• Gave presentation explaining the project
• Sent document to all

14. CHANGE VISION
• 26 page document
• Took over a month to get
alignment in version 0.1
• Sent to 20 senior
engineers for feedback
• Last version took into
account all feedback

15. COMMUNICAT E
THE VISION
• Monthly updates at all
hands meetings
• Weekly emails with
progress reports
• Events with food

16. EMPOWER BROA D - B A S E D ACT I O N

17. EMPOWER
ACTION
• The improvement backlog
was shared with everyone
• Some items prioritised by
voting
• Voluntary-based work
encouraged
• Test-a-thons and other
wide range initiatives

18. SHORT-TERM
WINS
• Used Scrum to guide the
change process
• Developed usable tools from
the beginning
• Developed one tool at a time
• Pushed adoption slowly until
tipping point
• Forced rest to adopt once
tipping point passed

19. INTEGRATION TIMES

20. 100.00%$
90.00%$
80.00%$
70.00%$
60.00%$
50.00%$
40.00%$
30.00%$
20.00%$
10.00%$
0.00%$
12/11/13$
12/18/13$
12/25/13$
1/15/14$
1/22/14$
1/29/14$
1/1/14$
1/8/14$
2/12/14$
2/19/14$
2/26/14$
2/5/14$
3/12/14$
3/19/14$
3/5/14$
TEST PLAN

21. PROJECT BURN DOWN

22. 100"
90"
80"
70"
60"
50"
40"
30"
20"
10"
0"
FROM 35 DAYS TO 15 DAYS
COMMIT AGE
Commit&to&Prod&Lag&
5/13/13" 6/13/13" 7/13/13" 8/13/13" 9/13/13" 10/13/13" 11/13/13" 12/13/13" 1/13/14" 2/13/14" 3/13/14" 4/13/14" 5/13/14"

23. I T ’ S A LWAYS HARDER THAN YOU THOUGHT
DON’T LET GO

24. LARGEST IMPEDIMENTS
• Technical
• Eat your own dog food
• Replicating the Sandbox
• Human
• People added to the team not sharing vision
• Getting teams to use the tools
• Managing expectations and communication

25. DON’T LET GO
• Fully dedicated team for a whole year.
• After the project ended the team remained as a new
unit.
• Even through the hardest times management gave full
support.

26. INCORPORATE CHANGE
INTO THE CULTURE

27. INCORPORATE CHANGE
INTO THE CULTURE
• Training program for new recruits
• Engineering partners are expected to conform to new
way of working
• Training program for engineering partners
• Tools in place become the cultural gates

28. OTHER TECHNIQUES WE USED
THERE IS MORE TO
CHANGE THAN KOTTER

29. WE ASKED THOSE ALREADY ONBOARD TO HELP OTHERS DO THE SAME
BRIDGE BUILDER

30. CHAMPION
SKEPTIC
Find someone senior in the
organisation:
• That wants the goal to be
reached
• That doesn’t fully support
your technical solution
• Talk with him regularly

31. DEDICATED
CHAMPION
YOU WON’T GO FAR
WITHOUT ONE

32. MAKE SURE TO BRING SNACKS TO GET TOGETHERS
DO FOOD

33. WILLING TO TRY THINGS BEFORE THEY ARE READY
EARLY ADOPTERS

34. LINKED IN WENT THROUGH THE SAME PROCESS
EXTERNAL
VAL IDATION

35. MAKE PEOPLE PROUD OF BEING PART OF THE CHANGE
GROUP IDENTITY

36. WE USED LARGE SCREENS TO SHOW PROGRESS
IN YOUR SPACE

37. IF YOU WANT LASTING CHANGE…
INVOLVE EVERYONE

38. AND WE DID IT ALL
USING SCRUM

39. “Never send a human to do a machine’s job”
-AGENT SMITH

40. REFERENCES
• Leading Change - John P. Kotter
• Fearless Change - Linda Rising, PhD
• Continous Delivery - Jez Humble, David Farley