A look at some of the methodologies that have shaped the direction of agile software development. We take a look at Lean Software Development (and the Toyota Production System), the Theory of Constraints and Systems Thinking.
Disclaimer: I am not and expert in any of the fields. Feel free to shout up at any point if there are any questions.Agile can sometimes feel like a cult, especially if it is blindly followed without questioning why.We talk a lot about the technical practices but often neglect to understand the management theory behind agile.This is not just for 'management'. It is relevant to all people involved in an agile organisation.- It is about setting the context for the things we do
Before:Software development is a very young field, so it isn't surprising that the prevailing ideas come from the field of manufacturing. What might be surprising is that some of the underlying ideas are fairly old.It is also worth remembering that although some of the ideas we are going to talk about have their roots in manufacturing, these ideas are not about manufacturing. They are equally applicable in knowledge work.This is not a complete history, I am only focusing on a few key people and ideas. There are many more.Prime example being Ford’s Model T production lineSee also Theory X and Theory Y
Before:The early 20th century starts to see investigations into alternative ways of working. A couple of names will appear again and again; TaiichiOhno and William Edwards Deming, who will see more of later...
JIT => Producing cars at the rate of customer demandTPS is the source of most of the agile related Japanese works you will hear banded around
Can you see some agile themes emerging here?TPS allowed Toyota to enjoy a lot of success, so this has to make a massive impact in other organisations, right? Nope. Largely ignored in Japan until external conditions (oil crisis) ended the boom of the 60’s. Ignore by the rest of the world for a lot longer.
Whilst Toyota was thinking about waste, Eli Goldratt was thinking about Constraints
EliGoldratt wrote the book whilst working at a software company making software for mapping manufacturing processesPeople like the book more than the softwareHe was sacked. Founded a TOC institute instead
1) Identify the constraint*How do we identify the constraint?*- Often work piles up in front of it. Scrum boards make it visible2) Decide how to exploit the constraint*What in agile allows us make this kind of decision* Retrospectives give teams chance to discuss3) Subordinate everything else to the constrain - align to the constraint, protect with WIP limits*maybe protect with WIP limits4) Elevate the constraint - increase the constraints capacity*How?* Change roles within teams - generalisation over specification, swarming etc5) If the constraint has been broken identify the new constraint and start again.
Although TOC has some salient points, it never mapped easily to software development. Many of the ideas have been further evolved into the concepts around kanban and continuous flow.It is no accident that one of the key people people in the introduction of kanban into SD (David Anderson) was also a driving force behind the adoption of the ideas of TOC into SD
Blindly copying the tool and practices is unlikely to lead to good resultsEventually we arrive at Lean Software Development - based on the principles of Lean
Principles relate closely to the original Lean principles1) Eliminate waste - identify it, remove anything that does not add customer value*What are the main forms of waste in SD?* Partially done work, requirements churn, un-required features*How do we avoid churn and un needed features?* Specify stories just in time2) Build quality in - not test it in later, don't create the defects in the first place.*How?* TDD, Acceptance testing, CI
3) Create knowledge - base it on facts from feedback- feedback comes from building software, not just specifying it, release early (CD?), generate fast feedback (tests)4) Defer commitment - make irreversible decisions at the last responsible moment, make decisions reversible or at least provide a framework to support changes- planning is not the same as making a commitment.*Flexible architectures, acceptance tests* e.g. NCMP5) Deliver fast - get working software to customers*Showcase demo, short iterations, small stories delivering value*
6) Respect people - Excellent leaders, expert workforce, self organisation*retrospectives, dojos, organisational learning*7) Optimize the whole - avoid the temptation to sub optimize. optimizing a subsystem will almost certainly sub optimise the whole*What situations apply to this?It is no good having an optimized development team if upstream and downstream teams are not geared up to work at the right cadence.*A good time to look at Systems Thinking
Systems thinking is a massive topic, so will focus on a key few people and ideas.
A set of things interconnected in such a way that produce their own pattern of behaviour over time.Has a function or purpose*but the purpose of a system is what it does, not what it says it does*It is possible to see the purpose of a system in different ways**What is the purpose of a supermarket? Supply products to customers? To generate a profit for owner? to provide employment?*What is the purpose of software development?Systems are powered by feedback. Behaviour emerges as a result of the inputs. *Feedback could be reinforcing e.g. bank interest, market share, birth rates, or balancing e.g. death rate,*Interactions are not linear - "Cause and effect are not closely related in time and space" - Peter SengePeople are good at seeing linear relationships, but we often get caught out by linear thinking, causation versus correlationWhat are the examples in SD when cause and effect are not closely related in time and space?- Technical debt- problems with analysis cause issues and rework later in the process- Team 'safety' (how safe do the team feel from blame and recrimination) effects estimation. Unsafe teams increase estimates and game metrics as a defense mechanism. Incentives do not always work as expected
*It is hard to separate out Deming's work into a separate field. It certainly influence TPS and TOC, and widely influences agile today*
This is part of a wider generalization that focusing to improve/reduce X will usually have the opposite effect.What does this look like in SD?- increasing output by skipping unit testing or dropping technical practices
1) Appreciation of system - understand the system as a whole*So it SD this is the entire value chain*2) Knowledge of variation - understand the variation in the system2 kindsCommon cause variation - The variation inherent in the system. The system noise.*What common cause variation exists in SD? - story and task estimation versus the time taken, velocityGenerally within knowable limits based on historic performancespecial cause - a signal of a new or emergent phenomenon. It is non historical and cannot be predicted. It is usually evidence of some underlying change in the system* illness, new team members, departmental restructuring**SD, and knowledge work in general is a high variability industry.*3) Theory of knowledge - how do we explain and share knowledge?4) Knowledge of psychology - how does human nature affect the system.*Understand incentives and coercion. Cognitive biases, Dunning Kruger effect, impostor syndrome**Culture**With the results he achieved in Japan he must have made a massive impact in the States. right? Nope. Largely ignore almost until his death in 1993.
1) Personal mastery - a spirit of continual learning and working towards a personal vision.2) Mental models understand how ingrained assumptions and generalizations influence how we see the world and act3) Build a shared vision - build genuine commitment rather than compliance4) Team learning through dialogue - suspend assumptions and learn together* Does this ever happen?* What does agile have to help with this? Retrospective - prime directive?5) Systems thinking - The fifth discipline that integrated the other 4*Again, culture is a key part*
*He came up with some snappy laws as well**There is loads more - Systems Thinking deserves its own Dojo*
It is not possible to mandate performance outside of the capability of the system- Need to change the underlying culture and conditions and the performance will emergeA bad system will defeat a good people every time*Deming 95%* discussNeed to stop seeing the world as linear relationships and think more about underlying interactionsUnderstand the demand on the systems*John Seddon's concept of demand - 2 types failure demand and value demand**What is the failure demand on SD?* Rework, requirements churn - like the waste in Lean SD*You are not your job role, you are not a column on a scrum board
There are lots more ideas an people who have shaped agile
The foundations of agile
THE FOUNDATIONS OF AGILELean Software DevelopmentTheory of ConstraintsSystems Thinking
What is agile?How would we describe agile?• A set of patterns, practices and tools• Part technical practices• Part management theory• Part cult?
The prevailing mindset...• All about the division of labour• Adam Smith‟s pin factory• Frederick Winslow Taylor‟s Scientific Management• Interchangeable part and interchangeable labour• Little respect for workers, assumes they are lazy and want todo bare minimum• Grim for the workers
The Toyota Production System• Vision is to have just in time assembly• Aim to remove• Overburden (muri)• pushing people or machines beyond their limits• inconsistency (mura)• build quality into the system to avoid inconsistency• waste (muda)• 7 wastes ; over production, waiting, transportation, processing, inventories /WIP, movement, defects• Allowed TPS to reduce lead time and cost whilst improvingquality
TPS Principles• Continuous improvement - Kaizen• Respect for people - build trust, stimulate professionaland personal growth• Long Term philosophy, even at the expense of short termcapacity• Right Process, right results - "pull" work to avoidoverproduction, stop the line mentality, visualise problems• Develop people and partners - develop people who followyour companies mentality• Drive organisational learning through solving rootproblems
Theory Of Constraints“a chain is no stronger than its weakest link”• Introduced by Eli Goldratt in 1984 book „The Goal‟• The goal of a systems is always limited by at least oneconstraint, and increasing the flow through the constraintwill increase throughput
5 focusing steps1. Identify the constraint2. Decide how to exploit the constraint3. Subordinate everything else to the constrain -align to the constraint4. Elevate the constraint - increase the constraintscapacity5. If the constraint has been broken identify thenew constraint and start again
The key points from TOC• Local optimisation may not (most probably will not) lead tooptimisation of the whole.• It is not necessary to have maximum efficiency from nonconstraints.Some things it shares with TPS• Continuous improvements through employeeempowerment• Cultural transformation• Global rather than local goals
Lean• When the consultants got hold of TPS in the late 80 / 90sthere was no stopping it• Many tried to replicate, but championing the tools andprocesses without mindset and principles often failed• Lean was adapted to many different sectors• Lean manufacturing, Lean operations, Lean supply chain, Leanproduct development etc etc.
Lean Software Development• Established by Mary and Tom Poppendieck in 2003Has 7 principles1) Eliminate waste - identify it, remove anything that doesnot add customer value2) Build quality in - not test it in later, dont create thedefects in the first place
Principles (continued)3) Create knowledge - base it on facts from feedback4) Defer commitment - make irreversible decisions at thelast responsible moment, make decisions reversible or atleast provide a framework to support changes5) Deliver fast - get working software to customers
Principles (continued)6) Respect people - Excellent leaders, expert workforce,self organisation7) Optimize the whole - avoid the temptation to suboptimize. optimizing a subsystem will almost certainly suboptimise the whole
What is a System• Most things are a system or part of a system• A set of things interconnected in such a way that producetheir own pattern of behaviour over time.• Has a function or purpose• Can be very simple or very complicated• Systems are powered by feedback. Behaviour emergesas a result of the inputs• Interactions are not linear - "Cause and effect are notclosely related in time and space" - Peter Senge
William Edwards Deming• Born at the height of Taylorism, Deming studied statisticalprocess techniques• He took his statistical process control methods to Japanafter WW2 to high acclaim and success
Deming• The aim is to improve quality in order to reduce expenseand increase productivity.• As quality increases costs will fall• But, focusing on reducing costs will cause costs to rise• Created the PDCA cycle
Deming‟s system of profound knowledge1. Appreciation of system - understand the system as awhole2. Knowledge of variation - understand the variation in thesystem3. Theory of knowledge - how do we explain and shareknowledge?4. Knowledge of psychology - how does human natureaffect the system
Peter Senge• Took earlier systems thinking ideas to the create the concept ofa "learning organization“At its heart are 5 disciplines1. Personal mastery - a spirit of continual learning and workingtowards a personal vision2. Mental models understand how ingrained assumptions andgeneralizations influence how we see the world and act3. Build a shared vision - build genuine commitment rather thancompliance4. Team learning through dialogue - suspend assumptions andlearn together5. Systems thinking - The fifth discipline that integrated theother 4
Peter Senge‟s Laws• Todays problems come from yesterdays "solutions."• The harder you push, the harder the system pushes back.• Behaviour grows better before it grows worse.• The easy way out usually leads back in.• The cure can be worse than the disease.• Faster is slower.• Cause and effect are not closely related in time and space.• Small changes can produce big results...but the areas ofhighest leverage are often the least obvious.• You can have your cake and eat it too …but not all at once.• Dividing an elephant in half does not produce two smallelephants.• There is no blame.
Final thought from Systems Thinking• It is not possible to mandate performance outside of thecapability of the system• A bad system will defeat a good people every time• Need to stop seeing the world as linear relationships andthink more about underlying interactions• Understand the demand on the systems• You are not your job role, you are not a column on ascrum board
Further reading• Deming - Out of the crisis (maybe dont start with thisone)• Mary and Tom Poppendieck - Lean software development• Donella Meadows - Thinking in systems• Peter Senge - The Fifth Discipline• Eli Goldratt - The Goal• Tons more stuff out there