How to apply The Toyota Way to the continuous crafting of embedded software? Find out in Yves Caseau's presentation. Watch the video of his presentation here: http://www.youtube.com/watch?v=2-vDMYheb_E More Lean IT presentations and videos on www.lean-it-summit.com

1. Copyright © Institut Lean France 2013
3 & 4 October, 2013
Paris, France
European Lean IT Summit 2013
Lean Software Factory
Yves Caseau – Bouygues Telecom

2. Lean Software Factory –
Applying The Toyota Way to the continuous
crafting of embedded evolving software
October 4th, 2013 – v0.1
European Lean IT Summit
Yves Caseau, EVP New Products & Innovation
Bouygues Telecom
Yves CASEAU – October 2013 – Lean Software Factory
2/24

3. Outline

First Part:
Motivations for our “Bbox” Software Factory
Agile, Extreme Programming & Lean Software


Second Part:
Lean Software Factory (LSF) with Four Practices
Third Part:
2012 – 2013 : Lessons learned
Yves CASEAU – October 2013 – Lean Software Factory
3/24

4. Part 1: Motivations
1
Overall Corporate Goals - 2011
1.
Efficiency
•
•
•
2.
Agility
•
•
•
3.
Reduce development costs
Less « rework », faster (functional) convergence
Better Quality of Experience through better code
Reduce TTM, remove lost time, less waiting
Early stakeholders (Marketing) integration into development cycle
Continuous innovation flow
Capitalize vs. Turnover
•
•
•
Give to everyone an opportunity
to contribute
Satisfaction derived from
individual excellence
Pride (collective & corporate)
Yves CASEAU – October 2013 – Lean Software Factory
4/24

5. Part 1: Motivations
1
What defines a good software product ?
Generic goals:

Modularity
“degree to which a system’s components may be separated and recombined” (Wikipedia) … or the
capacity of the architecture to maximize in its decomposition the independence of subcomponents

Evolvability
“the property of having many abilities”, that is software that can serve many purposes, together with “the
capacity of adaptive evolution”

Openness
expose API, open source (scrutiny), platform
Specific to GW/STB products:



HW/SW interface : HW changes frequently + instable (protocols)
Embedded Linux
Legacy Assets (older “boxes”)
Our « modular middleware » ambition:



TTM / Lifecycle control
Modularity across HW
Open to external innovation
Yves CASEAU – October 2013 – Lean Software Factory
5/24

6. Five Years of Software Strategy (2011)

Part 1:
Motivations
Part 1: Motivations
1
2010
2011
2012
2013
2014
Deploy
Improve
Delight
• Beginning of SW
deployment on
legacy box
• QoE tuning
• Open API – First
Hackathons
• Open Innovation
•
Jinni
•
Ijenko
• Next Gen
Hardware
• Additional
Features
•Open Innovation
(followed)
11 Mai
Plan
• Partner
selection
• unified MW
Proof-ofconcept
• SW Factory
outline
• Continuous
improvement
Build
• Factory starts
• Unified &
Modular
Midelware
• Bbox Sensation
development
• Continuous
improvement
• Bbox sensation
launch on June
18th (on time)
• Three separate
products
• New UI
• Cloud Gaming
LSF I
Yves CASEAU – October 2013 – Lean Software Factory
LSF II
6/24

7. Part 1: Motivations
1
Software Factory

Automation



Industrial Tools & Practices



Build
Test
Configuration & Source version Management
TQM & Continuous integration
Value the development process as much as produced software

Agility / Quality / Openness
code
process
data
processors
Information Technology
Yves CASEAU – October 2013 – Lean Software Factory
practices
models
Continuous Deployment
users
Information Systems
ops
Continuous
Feedback
stakeholders process
dev
Software Factory
7/24

8. 5.
Extreme
Programming
6.
Small teams
Small batches
Time boxing
Coevolution of
code/design/architecture
Role of face-to-face
communication
User stories
1.
2.
3.
Test-driven
development
Sustainable pace
Code is valuable
Yves CASEAU – October 2013 – Lean Software Factory
1.
2.
3.
Toyota
1.
2.
3.
4.
SCRUM
Agile, Scrum & Lean
Agile
Part 1: Motivations
1
Visual Management
Practices and Rites
Reflection
1.
2.
3.
Kanban
Kaizen
5S and waste
removal
8/24

9. Agile
Motivations: Agile & Extreme Programming
1.
2.
3.
4.
5.
6.
Small teams
Small batches
Time boxing
Coevolution of
code/design/architecture
Role of face-to-face
communication :
User stories
Stakeholders
alignment
Extreme
Programming
Part 1: Motivations
1
1.
2.
3.
Test-driven
development
Sustainable pace
Code is valuable
Yves CASEAU – October 2013 – Lean Software Factory
Axioms:
- smaller pieces = less risk
- smaller pieces for faster adjustment
- Common rhythms to stay synchronized
Most dramatic change in our Taylorinspired large-scale organizations 
Oldest, most durable problem
(silos) + 100+ people-projects
Axioms:
(1)Innovation → agility & iteration
(2)Agility → test velocity
End-to-end testing is difficult !
Still too much overload … and
exhaustion
To hurry => to err 
9/24

10. Motivation: Scrum & Lean
SCRUM
Efficient communication channel
- leverage body language
- avoid redundancy
- foster collaboration
1.
2.
3.
Visual Management
Practices and Rites
Reflection
Cf. Aristotle:
We are what we repeatedly do – Excellence, then, is not
an act but a habit
Axiom:
Hyper-competition & complexity ⇒
(excellence ⇒ continuous improvement)
Toyota
Part 1: Motivations
1
1.
2.
3.
Kanban
Kaizen
5S and waste
removal
The main symptom of dysfunctioning was (2011) and
still is (2013) …waiting for one another
Team problem solving is the best training tool … to
understand the complexity of our own systems !
The only way to do faster and better is to do
less 
Yves CASEAU – October 2013 – Lean Software Factory
10/24

11. Part II

First Part:
Motivations for our Bbox Software Factory
Agile, Extreme Programming & Lean Software


Second Part:
LSF with Four Practices
Third Part:
2012 – 2013 : Lessons learned
Yves CASEAU – October 2013 – Lean Software Factory
11/24

12. Part II: LSF Practices
2
The Art of Lean Software Development

Practice 0: Source Code Management and
Scripted Build
Practice 1: Automated Testing
Practice 2: Continuous Integration

Practice 3 : Less Code







Already deployed in
our IT software
development center
(Nantes)
Prioritized requirements – YAGNI (You ain’t gonna need it )
BDUF (Big Design Up Front) – Avoid complexity
Reuse, coding standards, design patterns, …
Practice 4: Short Iterations
Practice 5: Customer participation
Cf. Architecture’s role …
Cf. SCRUM
« Customer Participation is a two-way street »
Yves CASEAU – October 2013 – Lean Software Factory
12/24

13. LSF Practices (1) – Team Problem Solving
Part II: LSF Practices
2
Cf. ITIL : problem ≠ incident
1.
Search for root causes, using
the « Five Whys » approach
team work with all stakeholders
3.
Build a collective action plan
detailed: what will be done, by who, how and why …
4.
Regular check of the plan application and its
consequences
need for discipline & perseverance
5.
Why ?
Visual description of the problem
a drawing is worth a thousands words
2.
LSF start:
May 2012
Since we work on systems, with long causal
chaines and feedback loops.
Since we need to fix causes and not symptoms …
To avoid repeating the same mistakes
All viewpoints are required to find the best
solutions
Buy-in and empowerment from all
Most often the problem vanishes or evolves by
itself since environment conditions have changed
Trace those four steps: A3-like document
Yves CASEAU – October 2013 – Lean Software Factory
13/24

14. LSF Practices (2) – Project Room
Part II: LSF Practices
2
1.
Visualize planning and milestones
Rite: Sprint Planning Meeting
2.
LSF start:
May 2012
Why ?
To stay synchronized,
to avoid waiting
Visualize « workflow » (WBS)
see the process and the succession of steps
•
Multi-scale if needed
3.
Visualize « issues »
•
•
Display the problem-related A3
List of most important incidents « GdM »
Understand « the big picture » and
facilitate transitions
All viewpoints are required to find the best
solutions
Buy-in and empowerment from all
A place for collective memory, hence reflection 
4.
•
•
SCRUM rite after each sprint
Experience Return after each project
Yves CASEAU – October 2013 – Lean Software Factory
Learning and
continuous
improvement
14/24

15. Part II: LSF Practices
2
LSF start:
May 2012
LSF Practices (3) – Reduce WIP
1.
Visualize all tasks assigned to teams
i.e.: to place post-its into cells
2.
Reduce WIP (Work in Process)
Add constraints to the work load
•
No more than X post-its per cell 
3.
JIT management (Pull flows)
•
•
Avoir overload, multi-tasking and stockpiling
« work waiting to be handled »
In an ideal project setting, with an optimal scheduling,
there is no difference between push & pull
« push » happens when the end of step N activity signals
the beginning of step N+1 activity (most often with some
delay compared to the optimal schedule)
The more optimized the schedule, the more delays are amplified
•
•
« pull » means that step N – 1 production is governed
by the capacity of step N, which is more robust
(for instance, design vs. development)
Each post-it is a signal (Kanban)
Only works with small batches
Yves CASEAU – October 2013 – Lean Software Factory
Avoid « useless work » (muda)
• waiting code
• unused features
• minimize reponsability
handovers
• avoid setup times necessary to
switch from one type of activity to
another
15/24

16. Part II: LSF Practices
2
LSF start:
May 2012
LSF Practices (4) – Love Your Code
1.
Sort, organize and structure code (cf. Practice 0)
Cf. « 5S with Java » - p. 192 from
Poppendieck
Lean : 5S (Sort, Systematize, Shine, Standardize, Sustain)
2.
Discipline (coding styles & rules)
Work better, more efficiently
Collaboration & maintenance
Define guidelines and automate checking
3.
Code reviews & « elegant programming »
Display you code with pride to your colleagues
4.
Less code (cf. Hibb)
•
•
5.
Remove what is no longer in use
Avoid what will not be used much
« Gardening » (code refactoring)
•
•
code is alive (life  recycle )
Iterative process leads to accumulation
Yves CASEAU – October 2013 – Lean Software Factory
Code quality and Experience quality are
linked to one another
Collaboration & Capitalization
Future is uncertain, hence favor agility
over expressiveness
Cf. Google :
50% of code changes every month
Avoir accumulation, since it generates
costs & complexity
16/24

17. Part II: LSF Practices
2
LSF Deployment

Factory



What worked better/faster than expected:




Standups meetings (reduce email)
User stories (key to facilitate product owner’s role)
Visual Management (related to SCRUM)
What worked more slowly than expected:



Code management tools (IBM RTC) – 100% useful, even if it generates lots of debates 
Automated SW integration test (wall of boxes)
“All hands on deck” , Pull versus push (still too much waiting)
Unit testing (test-driven development)
What is hard:


Reflection & Slowing down
Large-scale orchestration
Yves CASEAU – October 2013 – Lean Software Factory
17/24

18. Part III

First Part:
Motivations for our Bbox Software Factory
Agile, Extreme Programming & Lean Software


Second Part:
LSF with Four Practices
Third Part:
2012 – 2013 : Lessons learned
Yves CASEAU – October 2013 – Lean Software Factory
18/24

19. Agile versus « V » Development Cycles
Part III: Lessons learned
3
Not everything is « agile » …
Clear & stable requirements, homogenerous code (e.g., technical patterns), Service Platform
integration ⇒ V cycle (strength of engineering, design methods, abstraction and anticipation)
 Unclear / Unstable requirements ⇒ agile
 Hardware projects (costly integration) ⇒ engineering + agile prototypes

System
Engineering
Ménadier’s “W cycle”
Iterative design
Time-boxing
Small teams
Project Room
(time & location)
Etc.
•
System
Integration
Coordination
Lean Factory:
continous integration
& deployment
Evolving existing components /
Build new (light) ones
Building new “heavy & stable” components
… but almost all activities benefit from a « lean » approach.
Cf. key ideas from Toyota Product Design:
•
Set-based design – parallel exploration, delay design choices with the most consequences
•
Tight flow (no place for problems to hide )
Yves CASEAU – October 2013 – Lean Software Factory
19/24

20. Part III: Lessons learned
3
Lean & Architecture (1)

Agile or lean software development does not prevent from building complex
products




In a continuous & incremental development process, one needs
an architectural framework to prevent from diverging



Complexity requires sense & common vision
Architecture is about communication & story telling
Architects are one of the backlog stakeholders
Refactoring is not enough (too much rework)
Architect must work in “pull mode” – they are here to assist
developers (not the other way around !)
A key Toyota principle is to share a systemic vision between all
process/product actors


Visual management → well-crafted artifacts
Design reviews are critical
Yves CASEAU – October 2013 – Lean Software Factory
20/24

21. Part III: Lessons learned
3
Lean Architecture (2)

Lean ≠ Agile, truly complement each other



Architects are needed to reconcile
« small scale » & « large scale » visions




Agile: well suited to change, focus on present,
delay decisions to match environment
Lean : well suited to complex/ large-scale,
focus on future & long-term, anticipation is welcome
Building a « platform approach »
Architecture is the « grammar for cooperation »
Conway’s law: (Architecture ⇔ organisation)
Coplien & Bjørnvig:
 You cannot always refactor your way
to a better architecture.


The essence of Lean Architecture is to take careful,
well-considered analysis and distill it into APIs 
Remember that architecture is mainly about people
Yves CASEAU – October 2013 – Lean Software Factory
21/24

22. Part III: Lessons learned
3
LSF Gouvernance

LSF Steering Committee






Support System
Tools
Engineering, Architecture
Methods et Project Management
Community
Scrums Masters community




Steering Committee
LSF Support System


Every two months, to re-align vision & goals
A moment to dissent & disagree (makes change leader stronger)
Educate managers about « change management »
2.0 (share & exchange) practice to capitalize
Learn from other SCRUM-practicing divisions (IT, digital)
Get inspiration from agile communities in other companies
Values : 10 self-inspired principles
Yves CASEAU – October 2013 – Lean Software Factory
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
Customer Focused
Self-empowered teams »
Time Boxing
Less is more
Software Pride
Dare to innovate
Humble listeners
Continuous story tellers
Knowledge is worth sharing »
« Respect pleasure »
22/24

23. 3
Gemba Walks
Part III: Lessons learned



1.
2.
3.
Twice a week, one hour per visit
On-going, learning process 
Build a rite = follow the same « script » each time
I should have listened to
Michael Ballé and started
sooner 
Show me what you do
show me your code / your product / your demo / your design
individual or group
Tell me why your are doing it this way.
This is the opportunity to share the vision 
What are your current problems ? Who are your stakeholders ?
Seven tips for a healthy ‘Gemba Walk’ / MBWA
Karmona Pragmatic Blog
 Visit everyone
 Go alone – Daily standup meetings aren’t enough
 Don’t bypass middle management e.g. don’t change priorities, requirements or design
 Observe, ask and LISTEN
 Be genuine, have fun and strive to catch your engineers doing something right and not something wrong (you
are not the “fun-police” ;)
 Share your dreams and vision
 Don’t “disturb” the Gemba – Timing is everything…
Yves CASEAU – October 2013 – Lean Software Factory
23/24

24. Part III: Lessons learned
3
Reflection : A Moment of Truth 


 Lauched Bbox sensation on time
 Approximately as many bugs as similar products,
longer than expected to stabilize
Too long to reach
desired stability
SW mastery


SW/HW coupling
Poor delivery time
forecasting
Slow delivery of
improvements
Skills to detect earlier & better
understand
Lack of experience + pressure
from stakeholders 
Pressure → breaks the “small
batches” principle 
Sucess is mostly a matter of skills !
The good news is that we learned a lot, the bad news is that we did not know enough 
A key methodological difficulty is that it is still hard to forecast how long it will take to
solve a problem
→ Stakeholder solidarity is still crucial 
Yves CASEAU – October 2013 – Lean Software Factory
24/24

25. Conclusion
LSF : a « new » vision
about software
products
•
•
•
•
LSF works 
•
•
•
Values are everything
•
The way you build is as important as what you build
… SW is a “live object”, constantly evolving
→ lean is a must, Taylorism does not work
“best-of-breed integration” of agile SW practices
Consumer Electronics Products require extremely short
development cycles
Focus on skills, what matters in the end
SW production/delivery automation is a must for embedded
SW products
Lean (Toyota-style) maximizes motivation
Learning is a satisfaction growth engine
(Toyota Kata) Practices ! Learn by doing …
Yves CASEAU – October 2013 – Lean Software Factory
25/24

26. Copyright © Institut Lean France 2013
3 & 4 October, 2013
Paris, France
More Lean IT presentations and videos on
www.lean-it-summit.com