A story of reducing technical debt
Founder + Digital Craftsman
Markus Kobler
Avoiding Technical Bankruptcy
Overview
The technical debt build up
Five Techniques that Worked
1) Getting a Canary
2) Rejecting Scrum for Kanban
3) Plan...
Taglab did a bit of everything web related...
Strategy
Transactional Sites
Design
Email
(Even built/hosted our own campaig...
Notable Clients Included
Betfair
Times Online
Starbucks
The Economist
Disney Channel
Turner
Global Switch
Channel Five
NBC...
My First three years at Taglab had been exciting
- Six fold growth
- Wide range of big and small projects
- Rounded skills...
Over time technical debt started to build up
Its around this time Taglab asked me to rejoin the company as CTO...
Supporting 40-50 live production web apps
- 100+ Environments in total (UAT, Integration, etc)
- Key required 24/7/365 SLA...
Pain stemmed from multiple points
Legacy and modern Codebase
- 7+ year old code, Singleton galore with poor test coverage
...
Projects were increasingly Death Marches
Death March TypesHappiness
Chances of Success
Kamikaze Mission Impossible
UglySuicide
Team are sacrificial
lambs at the sla...
At its core this is a story aboutHappiness
Chances of Success
Positively and creatively changing
peoples perspective to im...
Problem 1 : Notification Hell
Nagios monitoring was already in place
- but needed updating
Were getting 4-5 alerts a day!
...
Getting a Canary (or Nabaztag)
Flickr: julianbleecker
Flickr: krissatmontreal
Could differentiate personal
from work txt messages
Problems started feeling a
little less ‘catast...
Problem 2 : Juggling Delivery + Support
We where hungry for work, so new projects varied
wildly in size and complexity
- S...
'Known Knowns' - requirements captured in the specification
'Known Unknowns' - requirements flagged as ‘assumptions’
'Unknow...
Sounds like Agile Right?
Many competitors and clients were touting SCRUM
“Don’t you know the difference between the Chicke...
Rejecting Scrum for Kanban
To-Do Doing Done
Task Task
Task Task
Task
Task
Task
Task
Task
Task
TaskTask
Task
So we started ...
Rejecting Scrum for Kanban
To-Do Doing
Task Task
Task
Task
Task
Task
Task
Task
Task
Task
Task
Task
Task
Added columns that...
Rejecting Scrum for Kanban
Backlog In Progress
Task Task Task
Task
Task
Task
Task
Task
Task
Task
Task
Task
Task
Added colu...
Rejecting Scrum for Kanban
Backlog In Progress
Task Task
Task
Task
Task Task
Task
Task
Task
Task
Task
Added swim lanes
UAT...
Rejecting Scrum for Kanban
Backlog In Progress
Task Task
Task
Task
Task Task
Task
Task
Task
Task
Task
Introduced Work In P...
Issue
Improv-
ement
Improv-
ement
Rejecting Scrum for Kanban
Backlog In Progress
Task Task
Task
Task
Task Task
Task
Task
T...
Rejecting Scrum for Kanban
Work is continuously pulled through system
Visually provides a Micro & Macro view of work
Suppo...
Problem 3 : Estimation + Tracking Velocity
Flickr : kozumel
Planning Poker
Cards going up in value
User Story discussed
Once happy, everyone
estimates in private
Reveal at same time
...
Shared ownership of projects problem solving
Could start analytically looking back at projects
past progress
Enabled plott...
Problem 4 : Deployment
Deployment should be the last problem
developers need to worry about
- Needed to re-instil confidenc...
Designers Needed Something Simple
Flickr : thunderchild5
Developers Demanded More Rigour
Flickr : shutupyourface
Automating Deployments with ‘Munge’
Mix of automation and strict project conventions
- Minor changes could be released in ...
Problem 5 : Shared Ownership of Prod Issues
Whole team needed to share ownership of
dealing with production issues
Most pr...
So we got some hats of responsibility
Flickr : doctorow
So we got some hats of responsibility
Flickr : c_r_i_s
So we got some hats of responsibility
Flickr : striatic
So we got some hats of responsibility
Flickr : blackcountrymuseums
So we got some hats of responsibility
Finally a Success Story
Flickr : dybarber
Finally a success story
2009, Setanta Sports was on verge of collapse
- A key client dependant on their sports coverage
- ...
Success Story
Work broken into 5 interleaved dev streams
Hit first key 3 day deadline
- 4 major releases followed over next...
Final Thoughts
Our biggest challenge was turning bleak
situation into positive opportunities
- Some times you need a creat...
Credits
Death March (Yourdon Press Series) - http://www.amazon.co.uk/Death-March-Yourdon-Press-Edward/dp/
013143635X/ref=s...
Thank you.
distinctive.co
@markuskobler
Upcoming SlideShare
Loading in …5
×

Avoiding Technical Bankruptcy

696 views

Published on

A story of reducing technical debt

Published in: Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
696
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
13
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Avoiding Technical Bankruptcy

  1. 1. A story of reducing technical debt Founder + Digital Craftsman Markus Kobler Avoiding Technical Bankruptcy
  2. 2. Overview The technical debt build up Five Techniques that Worked 1) Getting a Canary 2) Rejecting Scrum for Kanban 3) Planning Poker 4) Automating Deployments with ‘Munge’ 5) Sharing Ownership with Fez & Pirate Hats! Dealing with Collapse : A Success Story
  3. 3. Taglab did a bit of everything web related... Strategy Transactional Sites Design Email (Even built/hosted our own campaign system!?!) Banners Micro-Sites Brochure-ware Social Media Hosting Ad Campaigns 24/7/365 SupportJ2EE Web Apps Messaging Middleware Workflow Systems
  4. 4. Notable Clients Included Betfair Times Online Starbucks The Economist Disney Channel Turner Global Switch Channel Five NBC Universal Skandia STA Travel Yell Online The Spectator British Gas Avis One Water Liverpool Victoria
  5. 5. My First three years at Taglab had been exciting - Six fold growth - Wide range of big and small projects - Rounded skillset from architectural & development to system administration However after the initial exciting growth the company started to hit a celling - Common in small business, 80% fail in their first 5 years* - J2EE was loosing its perceived competitive advantage *The E-Myth 2001
  6. 6. Over time technical debt started to build up Its around this time Taglab asked me to rejoin the company as CTO...
  7. 7. Supporting 40-50 live production web apps - 100+ Environments in total (UAT, Integration, etc) - Key required 24/7/365 SLA’s Hosted Apps had 45+ external integration points - RESTful & SOAP API’s - But many XML(sometimes)/RPC Designed by Architects with Capital A... ...but built by developers (with a lowercase d) Pain stemmed from multiple points
  8. 8. Pain stemmed from multiple points Legacy and modern Codebase - 7+ year old code, Singleton galore with poor test coverage - Some code even needing decompiling from bytecode... - To modern lightweight clean GWT, Hibernate, Spring Hosting Mix - Mix of our own rack in Docklands to clients hardware and the cloud - Although our rack provided many benefits some hardware dated back 5-7 years
  9. 9. Projects were increasingly Death Marches
  10. 10. Death March TypesHappiness Chances of Success Kamikaze Mission Impossible UglySuicide Team are sacrificial lambs at the slaughter for successful delivery Team dreams of fame, glory and riches ‘if’ they succeed Projects and Team are doomed. Only alternative is to be fired Projects are doomed, but everyone agrees it will be glorious failure! Death March - EdwardYourdon
  11. 11. At its core this is a story aboutHappiness Chances of Success Positively and creatively changing peoples perspective to improve delivery
  12. 12. Problem 1 : Notification Hell Nagios monitoring was already in place - but needed updating Were getting 4-5 alerts a day! - Regular Database/Hardware failures - Deployment, Code Quality problems - Subtle networking issues (ISP or self inflicted) You dreaded your phone Make things worse, we needed more checks! - CI still needed + much better test coverage
  13. 13. Getting a Canary (or Nabaztag) Flickr: julianbleecker
  14. 14. Flickr: krissatmontreal Could differentiate personal from work txt messages Problems started feeling a little less ‘catastrophic’ The office knew when was not right time ask for a quote on ‘one more copy change...’ Over time major alerts dropped to 1 every few weeks Getting a Canary (or Nabaztag)
  15. 15. Problem 2 : Juggling Delivery + Support We where hungry for work, so new projects varied wildly in size and complexity - Some were multi year builds - Others ‘OMG we need it done by the weekend’ Support’s unpredictability impacted new build work - More common in-practice with teams impacted by 2nd/3rd line support issues Mixing both functions can be efficient and satisfying
  16. 16. 'Known Knowns' - requirements captured in the specification 'Known Unknowns' - requirements flagged as ‘assumptions’ 'Unknown Knowns' - requirements the client had not yet told you about 'Unknowns Unknowns' - requirements neither you or client knew about Successful Delivery is about Accepting Uncertainty Flickr : ooocha
  17. 17. Sounds like Agile Right? Many competitors and clients were touting SCRUM “Don’t you know the difference between the Chicken and Pig!” But clients still found it hard to sign off budgets “Need to know what is being delivered before I signing off costs!” Work/Budget would often not fit neatly into an iteration(s) How would unpredictable support be incorporated? - Having a ‘batman’ would have been wasteful or not enough We needed something less rigid
  18. 18. Rejecting Scrum for Kanban To-Do Doing Done Task Task Task Task Task Task Task Task Task Task TaskTask Task So we started simply
  19. 19. Rejecting Scrum for Kanban To-Do Doing Task Task Task Task Task Task Task Task Task Task Task Task Task Added columns that made sense to us UAT CAT Live Task Task Task TaskTask
  20. 20. Rejecting Scrum for Kanban Backlog In Progress Task Task Task Task Task Task Task Task Task Task Task Task Task Added columns that matched our internal processes UAT CAT Live Task Task Task Task Task Planned Dev Internal Test Task Task Task Task Task Task Task Task
  21. 21. Rejecting Scrum for Kanban Backlog In Progress Task Task Task Task Task Task Task Task Task Task Task Added swim lanes UAT CAT Live Task Task Task Task Planned Dev Internal Test Task Task Task Task Task TaskTask M ajorProject1 M ajorProject2 OtherProjects Internal Task Task Task Task
  22. 22. Rejecting Scrum for Kanban Backlog In Progress Task Task Task Task Task Task Task Task Task Task Task Introduced Work In Progress (WIP) limits and SLA’s UAT CAT Live Task Task Task Task Planned Dev Internal Test Task Task Task Task Task TaskTask M ajorProject1 M ajorProject2 OtherProjects Internal Task Task Task Task 5 Day SLA 15 4 12
  23. 23. Issue Improv- ement Improv- ement Rejecting Scrum for Kanban Backlog In Progress Task Task Task Task Task Task Task Task Task Task Task Visually distinguish types of tasks UAT CAT Live Task Task Task Task Planned Dev Internal Test Task Task Task Task Task TaskTask M ajorProject1 M ajorProject2 OtherProjects Internal Task Task Task Task 5 Day SLA 15 4 12 Improv- ement Improv- ement Issue Issue
  24. 24. Rejecting Scrum for Kanban Work is continuously pulled through system Visually provides a Micro & Macro view of work Support issues easily absorbed WIP and SLA’s provided key indicators of quality and resource planning
  25. 25. Problem 3 : Estimation + Tracking Velocity Flickr : kozumel
  26. 26. Planning Poker Cards going up in value User Story discussed Once happy, everyone estimates in private Reveal at same time Hi & Lo estimates discuss differences Repeat estimate again if no consensus Flickr: lejoe
  27. 27. Shared ownership of projects problem solving Could start analytically looking back at projects past progress Enabled plotting projects into future based on previous tasks/estimates As a PM you can often get team to develop simplest solution and feel good about it! Planning Poker
  28. 28. Problem 4 : Deployment Deployment should be the last problem developers need to worry about - Needed to re-instil confidence back in the team Had to work well with Designers and Developers Needed to adapt to different environments - Our Rack, Client’s hardware, The cloud Reducing time to deploy would reduce our costs - Worst cases deployments took 5 hours!
  29. 29. Designers Needed Something Simple Flickr : thunderchild5
  30. 30. Developers Demanded More Rigour Flickr : shutupyourface
  31. 31. Automating Deployments with ‘Munge’ Mix of automation and strict project conventions - Minor changes could be released in seconds - Major releases, tested and deployed in 5-20mins (depending on changes) Clean build environment and version controlled released meant predictable builds and rollbacks Used live environment metrics to determine deployed hosts
  32. 32. Problem 5 : Shared Ownership of Prod Issues Whole team needed to share ownership of dealing with production issues Most problems only needed one person looking at it to begin with Ideally ownership should be visual to rest of team
  33. 33. So we got some hats of responsibility Flickr : doctorow
  34. 34. So we got some hats of responsibility Flickr : c_r_i_s
  35. 35. So we got some hats of responsibility Flickr : striatic
  36. 36. So we got some hats of responsibility Flickr : blackcountrymuseums
  37. 37. So we got some hats of responsibility
  38. 38. Finally a Success Story Flickr : dybarber
  39. 39. Finally a success story 2009, Setanta Sports was on verge of collapse - A key client dependant on their sports coverage - Unsure if someone would step in with finance - Or someone else would buy the rights Particularly complex CRM/Sales intergration - Estimated >5 weeks worth of effort - Failure to deliver by immovable deadline was to be very damaging for all parties involved Sign off received 3 days before changes were required
  40. 40. Success Story Work broken into 5 interleaved dev streams Hit first key 3 day deadline - 4 major releases followed over next 7 days - More followed in the subsequent weeks/months Sales in the days that followed dwarfed anything that came before with 0 downtime One of the most satisfying projects have been lucky enough to be involved in All thanks to hard prior work put in by the team
  41. 41. Final Thoughts Our biggest challenge was turning bleak situation into positive opportunities - Some times you need a creative solution - Continuously take educated technical risks Software delivery is an art-form - but backed by scientific approaches Embrace change and unpredictability - but always keep a clear guiding vision An informative workspace is key - Although is largely about ‘Psychological Manipulation’
  42. 42. Credits Death March (Yourdon Press Series) - http://www.amazon.co.uk/Death-March-Yourdon-Press-Edward/dp/ 013143635X/ref=sr_1_1?ie=UTF8&qid=1290856412&sr=8-1 Nabaztag - julianbleecker - http://www.flickr.com/photos/julianbleecker/156303245/ Nabaztag - krissatmontreal - http://www.flickr.com/photos/krissatmontreal/3027612184/ 061108-F-5586B-137 - ooocha - http://www.flickr.com/photos/ooocha/3051551020/ Poker Phase - kozumel - http://www.flickr.com/photos/kozumel/4063207100/ Planning poker! I've a straight flush! - lejoe - http://www.flickr.com/photos/lejoe/4553607341/ The Big Easy - thunderchild5 - http://www.flickr.com/photos/thunderchild5/2397161370/ Factory - Robotic Arms - shutupyourface - http://www.flickr.com/photos/shutupyourface/105779625/ Me in Ada's pirate hat, monorail queue, Walt Disney World, Orlando, Florida - doctorow - http://www.flickr.com/ photos/doctorow/2137686/ Fez / 87.365 - c_r_i_s - http://www.flickr.com/photos/c_r_i_s/2852508138/ past tense - striatic - http://www.flickr.com/photos/striatic/2418853122/ 1940s flying helmet , LC1995_15_1 - blackcountrymuseums - http://www.flickr.com/photos/ blackcountrymuseums/4970602229/ Fremantle Bridge Collapse, 22 July 1926 - dybarber - http://www.flickr.com/photos/dybarber/3461384691/
  43. 43. Thank you. distinctive.co @markuskobler

×