How to make a large agile project succeed? Presentation from Scan Agile 2013 conference based on two big projects. Real world problems and solutions.

1. ANTTI VIRTANEN
+358 44 507 0050
antti.virtanen@solita.fi
Pieces of the Big Agile Puzzle problems and solutions
11.11.2013

2. Agenda
1.
The context and background
1.
2.
3.
4.
5.
6.
2.
Fundamental principles to follow
1.
2.
3.
4.
Ammunition for ad hominem arguments
The projects we will talk about
Lean and agile
What is a big agile project (our context)
Some relevant human limitations
Things they didn‟t teach you at the university
The foundation you need
Some unsatisfactory solutions
Actual problems and applying the principles
QA and throwing stones

3. Who? What experience?

4. Short background
About me
Programmer and a jack of all trades for some time.
Hobbyist -> Professional developer -> Researcher -> Developer ->
Architect
About Solita
250 employees. Enough to make serious things happen.
Growing, succeeding, profitable, constantly improving.
Expert at making complex tailored business software.
Serious capacity also at BI/DW, integration, consulting and UX.

5. The projects (in TOP-3 for Solita)
ERP (2007-2010)
Kirre (2010-2013)
Multiple teams
Yes. Also multi-site.
Yes. (multi-site)
Long build times
Hours :-(
Semi-solved.
Quality pressures
Yes. Delayed much.
Extreme. Delayed a a bit.
DB tables (oper.)
Approx. 800
Approx. 300
Codebase LOC
About 1 million
About 300k
Parallel work?
No.
Yes,after a fashion.
Happy developers?
No.. :-(
Yes.. mostly
Complex domain?
No, but unfamiliar.
Yes. And unfamiliar.

6. More about the ERP project
Solita‟s “death march” project
Unrealistic timetable.
Arrogance on our part.
A Pyrrhic victory
We possess a certain amount of sisu.
There were no good books or
guides to follow.
My reason for being here today.
Picture: Sisukas Nainen @ youtube

7. Kirre, the second big one..
Kirre keeps track of all land property ownership in Finland.
Problems would make Big Money very angry.
Failure in this project would have been fatal to Solita as a company.
Keys to success
Sisu and stubborness again. Defeat does not exist in this dojo.
Less arrogance. Hired professional coaches.
Books had been written. Studied a lot.
Not brute-forcing. Reshaped “process” totally.
This feels like a real victory.
There were issues, but this is something I can be proud of.

8. Agile and lean.
“Lean architecture and Agile feature development are
not about working harder. They are about working
„smarter‟ in the academic or traditional computer
science senses of the word „smart.‟”
Coplien, Lean Architecture

9. Big project
What is different?
Why can‟t humans cope with it?
What new skills are required

10. What is a big project? How is it different?
Bigger system poses technical challenges
Longer build times, slower test runs, complex deployments
Too many details.
Some documentation is required.
People issues
Multiple teams -> management and communication challenge.
Treating people as resources does not work.
Training new people is seriously expensive.
Motivation will falter after three years with no production
deployment.

11. Human limitations
Some well-understood limits
Context-switching is not free.
Eight hours per day available for work. At most.
Communication is unreliable, unpredictable and time-consuming.
There are limits to handling complexity
Social limitations and issues
There is no ”team” of 25 people.
Team formation is tricky
We are quite irrational
Rational choice is an illusion
We are geared towards short-term profits and instant rewards.

12. Engineering is not enough
Programming
Marketing
Systems thinking
UI design
Psychology
Mathematics
Testing
Philosophy
Drama and acting
We value things on the left.. in a big project things on the right even more!

13. Fundamental principles
These are simple principles. But not easy.
Decoupling
Composition over monolithic ideas
Avoid up-front decisions
Separation of concerns
Creating MVPs

14. Parallelism is the paramount concern
Teams move simultaneously toward a shared goal
A team must have a subgoal (purpose)
Do not hinder with accidental obstacles
Pull value, don‟t push (be lean)
Single PO or synchronization is push.
Pushing does not truly scale.
Central authority is also a risk factor.

15. Avoid monoliths,
favor composition
You can build a huge monolith in parallel…
It‟s a design and engineering decision
Functional programming
OS products and frameworks often
Decoupling functional specifications in a moment..

16. Avoid up-front decisions
When is the ”last responsible moment”?
Who knows.. but choosing technology up-front is too early
Pushing the moment further away:
1.
2.
3.
Paper prototypes. (sketching function)
Initial boundaries and API (Form follows function)
Iterate and write some code. (live form)
(For this path I recommend Coplien‟s book Lean
Architecture.)

17. Separation of concerns
Avoid leaking implementation details
External API – UI – data model
Tool selection
A big project does have very special needs!
Automate everything (that DevOps thing). Composable tools.
Rethinking tools
Maven is a repository, not a build tool.
Jenkins is UI for cron, not a CI tool.

18. Aim for MVP at all levels
You absolutely must postpone features.
But which ones are nice-to-have ?
You need feedback early on
Create a minimal PoC implementation
Apply this to programming, design, specifications etc.
A ”big picture” is too big to see
The MVP/prototype makes it clear as a side-effect!

19. Be wary of easy solutions

20. ”Solutions” that don‟t address the fundamentals
Technological miracles
Decoupling and separation of concerns (SOA, REST , ESB)
Opinionated frameworks (Rails)
Management Tools
”Hiding” complexity (BPML, TOGAF, MDM)
Simplified recipes
You can‟t ”Teach Yourself Lean in 21 Days”! Preposterous!
To be fair, some of these have value..
But you really can’t solve a human problem with
technological magic tricks!

21. Principles in real world
(What did we actually learn?)
Conditions of greedy selection no longer hold for the backlog
Decoupling of functional specifications
Normal form in relational database considered harmful
Focus on results, not tasks
Teams around functionality, not layers and technology
Leadership is not management

22. How big is the elephant?
Three ways to search the problem space:
Depth-first (to measure complexity and risk)
Breadth-first (to get an overview)
Explorative mapping (black art, you may get interesting
findings)
I highly recommend balancing between all three
in a big project. Early on!

23. Backlog priorization
What is the most valuable item?
The most risky one?
-> reduce risk
The one with most direct value to end-user (lean maxim)?
-> customer value
Getting a core workflow done in it‟s entirety?
-> feedback and reduction of risk
In a big project it is a good to rethink the criteria.

24. Decoupling functional specs (case Kirre)
Three features: Work queue, search and application handling
Separate from the user‟s point of view
So parallel work comes naturally?
Why not? Some bad reasons
The database is a monolithic normal form design.
The dev tools do not support this.
The build pipeline tool doesn‟t allow it.
The specification team (the ”architects” or ”PO-team”) is a separate silo.
We had limited success decoupling these things. But it is
definitely the right direction to go!

25. The relational database posse is lacking
The RDBMS mindset (for waterfalls)
A static view of the end state.
The ”normal form” is by definition a monolith!
Relational model is not broken per se
But ”agile developer” is not served by the vendors.
Normal form doesn‟t deal with uncertainty and agile.
Big projects are especially vulnerable to these issues!
Part of the NoSQL hype is a result of this attitude.

26. All models are wrong. Some are useful.
(actually we retain normal form here)
Application
type
Application
group
Property
Decision
Application
Queue_vie
w
Queue
Assignee
UI for
queue
handling
Views are great!
• Early feedback
• Parallel development
• Hidden complexity
• Decoupling

27. Everyone. Must. Focus. On. Results.
Bad projects (tasks)
Good projects (goals)
The elite (results)
What is the next task today?
What are you going to work on
today?
What are you going to accomplish
today?
How many tasks left in the backlog?
Is the invoice integration ready?
Does the user have all essential
invoicing functionality?
No - when will it be ready?
No - when will it be deployed?
Are we doing what was specified in the
contract?
Are we heading to the right
direction?
Are we using the resources (time and
money) in the best possible way?
Did we bill all the hours?
Is the customer happy?
What is the most valuable thing to do
next?

28. The amazing Kirre project tool (data migration)
Team‟s purpose
Test plan
Short term plan
(Kanban style)
The purpose statement helped maintain focus and prioritize backlog.
backlog

29. About team organization
Tempting the dark side is
Integrations – back-end –UI…
Where‟s the end-user? Where‟s the use-case?
Ok, use-case/scenario based then?
You need that cross-functional team. Got one?
You cannot force team formation!
The use-case based view is the right way. But not easy.
After many changes we reinstated the integration team.

30. Leadershipment (case Kirre)
A single project manager, about twenty people
A bottleneck, not parallel! A risk!
When faced with a serious situation. (final deadline)
Divided the responsibility (leadership) to five individuals
No reports (or management duties) required
No holds barred, bare fist prison rules.. Just make it happen
Did it work? It was total pwnage and salvation of the project!
The Great Enablers
For each team: Autonomy – mastery - purpose
Support and coaching. A scrum master who was not part of the team.
Big picture priorization over teams. Together with everyone.

31. To “make it so”, important considerations
Voluntary.
If a sane person commits, it‟s a good sign.
You must delegate power and freedom too.
You need a manager who is not insecure!
Leader must have some respect from peers.
Leadership is based on ”social influence” by definition.
A title is not related to this.
It’s about getting things done.
It‟s not a ”tech lead” role.

32. Summary

33. To summarize.. How to make it alive..
Flexibility in scope, budget and timetables are required.
Also the customer must accept this and trust you.
Respect the issues related to size
Motivation, complexity, human limitations.
No single architect, PO, manager to make all decisions.
Do not ”manage” and control, lead instead
Good employees don‟t need ”management”
Get real cross-functional teams. Motivated & autonomous.

34. Stuff that victories are made of
Lean
Agile
Continuous innovation
Continuous flow
Continuous feedback
Use-cases in backlog
Parallel development
Continuous Delivery
End-user needs
Compositional design
Continuous
Integration
MVP first
Form follows function
Automated tests
Empowered teams
Realistic shared vision and plan (backlog)
Motivation
Transparency
Honesty
Trust

35. Resources and further reading
Even more books
Lean From The Trenches
Lean Architecture
Lean Thinking
Lean Software Development: An Agile Toolkit
The Toyota Production System
Rational Choice in an Uncertain World: The
Psychology of Judgment and Decision Making
Nonviolent Communication: A Language of Life
Systems Thinking

36. @SolitaOy
Antti Virtanen | antti.virtanen@solita.fi
THANK YOU.