Go Faster!
Tech, Process and Organization
Fred George
fredgeorge@acm.org
@fgeorge52
Copyright © 2015 by Fred George. All rights reserved. 1
Fred George
Copyright © 2015 by Fred George. All rights reserved. 3
Consulting Roles
• Change Agent
• Disruptor
• “Hand grenade I am throwing
into development”
• CTO describing Fred to his Vice
President
Industry “Hot” Topics
Copyright © 2015 by Fred George. All rights reserved. 4
MicroServices
Cloud
Docker
Dev/Ops Full-stack
developer
Agile
Lean startup
Cassandra
Event bus
MVP (minimum
viable product
Programmer
anarchy
No-SQL
One Hacker
Way
Why: New Problem Domains:
The Cynefin Framework
Copyright © 2015 by Fred George. All rights reserved. 5
Simple
ComplicatedComplex
Chaotic
Cause a Effect
Cause aaa Effect
Effect? Cause?
Cause ? Effect
Disorder
Why: Competition is coming
Copyright © 2015 by Fred George. All rights reserved. 6
• Enablement through technologies
• Cloud computing
• Ubiquitous, high bandwidth
• Languages (and supporting frameworks)
• Recognition of business opportunities
• Silicon Valley innovators as role models
• Accelerating business needs
• Few inhibitors for global competitors
• Reduction of entry barriers for niche competitors
• Micro-marketing to specific customer niches
How Fast Can You Go?
Copyright © 2015 by Fred George. All rights reserved. 7
Iteration Length in Agile Age
Copyright © 2015 by Fred George. All rights reserved. 8
1 mo.
20152000 2005 2010
3 wks.
2 wks.
1 wk.
1 day
XP
XP
XP
XP
SCRUM* *2-3 months
Project Delivery Cycles
Copyright © 2015 by Fred George. All rights reserved. 9
5 yrs.
1980 1990 2000
1 mon.
1 week
1 day
2010
1 year
ProjectDuration(log)
Waterfall
Waterfall
Agile
Agile
A2
A2
Waterfall
w OO
How Fast Can You Go?
Copyright © 2015 by Fred George. All rights reserved. 10
Deployment to Production Every 3.5 minutes
Inhibitors
Copyright © 2015 by Fred George. All rights reserved. 11
•Cloud exploitation
•Specialized databases
•New programming languages
•Open source frameworks (users and creators)
•Continuous releases
“Valley Tech”
Copyright © 2015 by Fred George. All rights reserved. 12
Hardware Lead Times
Copyright © 2015 by Fred George. All rights reserved. 13
6 mo.
1990 1995 2005
30 min.
1 week
1 day
2015
1 mo.
2000 2010
5 min.
5 sec.
Orderleadtimes(log)
Data Center
Virtual
Machines
Docker
Commercial
Cloud
Capacity planning?
Dedicated ops team?
Copyright © 2015 by Fred George. All rights reserved. 14
10
1
1000
100
100,0001 10 100 1000 10,000
Size (loc)
NumberofServices
MicroServices
Rails / Java Zone
SO
A
Workshop
Forward
Netflix
Services Impact
0MQ
0MQ 0MQ
0MQ 0MQ 0MQ
Copyright © 2015 by Fred George. All rights reserved. 15
Kafka
Service
Emerging: Event Bus
New Architecture Patterns
Copyright © 2015 by Fred George. All rights reserved. 16
Bus
Service
Service
Service
Need
Need
Need
Sol’n’s
Sol’n
•Variants easy
•Graceful degradation
Choose
Sol’n
Incremental Applications
Copyright © 2015 by Fred George. All rights reserved. 17
Event Bus
Rental
offers
Car
rent
Legacy
server
Brand
offers
Location
offers
Segmentation
Membership
CarRent.com Login
Sally
A B
Databases:
Holy Grail or Ball-and-Chain?
• Fortune 100 view:
• Entity-oriented; consistency essential
• As few as possible
• MicroService view:
• Shift to Event Bus, plus
• DB per MicroService (if persistence needed)
• Poly-glot (various NoSQL, SQL) + event buses
• Few (10%) writable; even fewer transactional
Copyright © 2015 by Fred George. All rights reserved. 18
Copyright © 2015 by Fred George. All rights reserved. 19
Rental
offers
Car
rent
Legacy
server
Brand
offers
Location
offers
Segmentation
Membership
CarRent.com Login
Sally
A B Rental
offers
Event
publishing
Solution
collecting
300ms
responder
Redis
Copyright © 2015 by Fred George. All rights reserved. 20
Rental
offers
Car
rent
Legacy
server
Brand
offers
Location
offers
Segmentation
Membership
CarRent.com Login
Sally
A B
Membership
enrichment
ETL Key-value
Open Source: Netflix
Copyright © 2015 by Fred George. All rights reserved. 21
Open Source: Docker
Copyright © 2015 by Fred George. All rights reserved. 22
Disruptive Technology:
Functional Languages
Copyright © 2015 by Fred George. All rights reserved. 23
•Middleware exploiting:
• Kafka written in Scala
• RabbitMQ written in Erlang
•Case study: MailOnline page rendering
• Legacy system: 130K loc Java/JSP’s
• Replacement: 4,000 loc Clojure
Mitigate Process Inhibitors
Copyright © 2015 by Fred George. All rights reserved. 24
Understand Your Problem
Copyright © 2015 by Fred George. All rights reserved. 25
Complex
Simple
Complicated
Chaotic
Cause a Effect
Cause aaa Effect
Effect? Cause?
Cause ? Effect
Disorder
Copyright © 2015 by Fred George. All rights reserved. 26
Requirements
Trying ideas
“Experimentation
drives Innovation”
New Vision of “What” to Build
Copyright © 2015 by Fred George. All rights reserved. 27
Task
Story
Feature
Project
Initiative
Co-Credit: Greg Reiser, ThoughtWorks
Re-think Interactions
• Bad metrics:
• Lines of code
• Development stages and milestones met
• Business success metrics:
• Sales, clicks, registrations, customer retention, etc.
Measure What Matters
Copyright © 2015 by Fred George. All rights reserved. 28
Mitigate Organization
Inhibitors
Copyright © 2015 by Fred George. All rights reserved. 29
Over-Specialization
Copyright © 2015 by Fred George. All rights reserved.
•Theory:
• Specialist are more productive
•Practice:
• Overhead of communication is under-estimated
• Unbalanced workload creates delays
30
Specialization Institutionalized
with Titles
Copyright © 2015 by Fred George. All rights reserved. 31
Case Study:
•50 IT professionals
•25+ titles
•0 people understanding projects
Solution: Fix the Titles
Copyright © 2015 by Fred George. All rights reserved. 32
Step 1: Define competence in key technologies
Strategic Technologies
• Ruby
• Java
• iOS
• Android
• DB (SQL and non-SQL)
• Testing
• OO Design
• Node.js
• HTML/CSS/JavaScript
• …
New Titles for New Projects
Copyright © 2015 by Fred George. All rights reserved. 33
Not competent in a
key technology
Competent in a key
technology
Expert in a key
technology
Competent in 5-7
key technologies
Expert in 3-5 key
technologies
“Fix the Furniture”*
Copyright © 2015 by Fred George. All rights reserved. 34*Kent Beck, XP Explained
Non-dedicated Leadership
Copyright © 2015 by Fred George. All rights reserved. 35
Bring Work to the Team…
Copyright © 2015 by Fred George. All rights reserved. 36
Not musical
chairs for each
small project
Copyright © 2015 by Fred George. All rights reserved. 37
MicroServices
Cloud
Docker
Dev/Ops Full-stack
developer
Agile
Lean startup
Cassandra
Event bus
MVP (minimum
viable product
Programmer
anarchy
No-SQL
Managerless
process
Copyright © 2015 by Fred George. All rights reserved. 38
MicroServices
Cloud
Docker
Dev/Ops Full-stack
developer
Agile
Lean startup
Cassandra
Event bus
MVP (minimum
viable product
Programmer
anarchy
No-SQL
Managerless
process
Project Delivery Cycles
Copyright © 2015 by Fred George. All rights reserved. 39
NRK
NRK
5 yrs.
1980 1990 2000
1 mon.
1 week
1 day
2010
1 year
ProjectDuration(log)
Waterfall
Waterfall
Agile
Agile
Waterfall
w OO
Acknowledgements
Copyright © 2015 by Fred George. All rights reserved. 40
•Neil Hutchinson – Founder of Forward
• Adopting a “Fail Fast” culture
•Carl Gaywood
• Former programmer who wanted to Go Faster
•Paul Ingles and Mike Jones
• Implementers of “Experimentation drives innovation”
More Information…
•Google:
• MicroService Architecture for videos
• MicroXchg 2015, 2016 conferences
• MicroService track in most conferences
• Programmer Anarchy for managerless teams
• Docker for latest container deployment
Copyright © 2015 by Fred George. All rights reserved. 41
Go Faster:
Tech, Process, & Organization
Fred George
fredgeorge@acm.org
@fgeorge52
Copyright © 2015 by Fred George. All rights reserved. 42

Go Faster - Remove Inhibitors to Rapid Innovation

  • 1.
    Go Faster! Tech, Processand Organization Fred George fredgeorge@acm.org @fgeorge52 Copyright © 2015 by Fred George. All rights reserved. 1
  • 2.
    Fred George Copyright ©2015 by Fred George. All rights reserved. 3 Consulting Roles • Change Agent • Disruptor • “Hand grenade I am throwing into development” • CTO describing Fred to his Vice President
  • 3.
    Industry “Hot” Topics Copyright© 2015 by Fred George. All rights reserved. 4 MicroServices Cloud Docker Dev/Ops Full-stack developer Agile Lean startup Cassandra Event bus MVP (minimum viable product Programmer anarchy No-SQL One Hacker Way
  • 4.
    Why: New ProblemDomains: The Cynefin Framework Copyright © 2015 by Fred George. All rights reserved. 5 Simple ComplicatedComplex Chaotic Cause a Effect Cause aaa Effect Effect? Cause? Cause ? Effect Disorder
  • 5.
    Why: Competition iscoming Copyright © 2015 by Fred George. All rights reserved. 6 • Enablement through technologies • Cloud computing • Ubiquitous, high bandwidth • Languages (and supporting frameworks) • Recognition of business opportunities • Silicon Valley innovators as role models • Accelerating business needs • Few inhibitors for global competitors • Reduction of entry barriers for niche competitors • Micro-marketing to specific customer niches
  • 6.
    How Fast CanYou Go? Copyright © 2015 by Fred George. All rights reserved. 7
  • 7.
    Iteration Length inAgile Age Copyright © 2015 by Fred George. All rights reserved. 8 1 mo. 20152000 2005 2010 3 wks. 2 wks. 1 wk. 1 day XP XP XP XP SCRUM* *2-3 months
  • 8.
    Project Delivery Cycles Copyright© 2015 by Fred George. All rights reserved. 9 5 yrs. 1980 1990 2000 1 mon. 1 week 1 day 2010 1 year ProjectDuration(log) Waterfall Waterfall Agile Agile A2 A2 Waterfall w OO
  • 9.
    How Fast CanYou Go? Copyright © 2015 by Fred George. All rights reserved. 10 Deployment to Production Every 3.5 minutes
  • 10.
    Inhibitors Copyright © 2015by Fred George. All rights reserved. 11
  • 11.
    •Cloud exploitation •Specialized databases •Newprogramming languages •Open source frameworks (users and creators) •Continuous releases “Valley Tech” Copyright © 2015 by Fred George. All rights reserved. 12
  • 12.
    Hardware Lead Times Copyright© 2015 by Fred George. All rights reserved. 13 6 mo. 1990 1995 2005 30 min. 1 week 1 day 2015 1 mo. 2000 2010 5 min. 5 sec. Orderleadtimes(log) Data Center Virtual Machines Docker Commercial Cloud Capacity planning? Dedicated ops team?
  • 13.
    Copyright © 2015by Fred George. All rights reserved. 14 10 1 1000 100 100,0001 10 100 1000 10,000 Size (loc) NumberofServices MicroServices Rails / Java Zone SO A Workshop Forward Netflix Services Impact
  • 14.
    0MQ 0MQ 0MQ 0MQ 0MQ0MQ Copyright © 2015 by Fred George. All rights reserved. 15 Kafka Service Emerging: Event Bus
  • 15.
    New Architecture Patterns Copyright© 2015 by Fred George. All rights reserved. 16 Bus Service Service Service Need Need Need Sol’n’s Sol’n •Variants easy •Graceful degradation Choose Sol’n
  • 16.
    Incremental Applications Copyright ©2015 by Fred George. All rights reserved. 17 Event Bus Rental offers Car rent Legacy server Brand offers Location offers Segmentation Membership CarRent.com Login Sally A B
  • 17.
    Databases: Holy Grail orBall-and-Chain? • Fortune 100 view: • Entity-oriented; consistency essential • As few as possible • MicroService view: • Shift to Event Bus, plus • DB per MicroService (if persistence needed) • Poly-glot (various NoSQL, SQL) + event buses • Few (10%) writable; even fewer transactional Copyright © 2015 by Fred George. All rights reserved. 18
  • 18.
    Copyright © 2015by Fred George. All rights reserved. 19 Rental offers Car rent Legacy server Brand offers Location offers Segmentation Membership CarRent.com Login Sally A B Rental offers Event publishing Solution collecting 300ms responder Redis
  • 19.
    Copyright © 2015by Fred George. All rights reserved. 20 Rental offers Car rent Legacy server Brand offers Location offers Segmentation Membership CarRent.com Login Sally A B Membership enrichment ETL Key-value
  • 20.
    Open Source: Netflix Copyright© 2015 by Fred George. All rights reserved. 21
  • 21.
    Open Source: Docker Copyright© 2015 by Fred George. All rights reserved. 22
  • 22.
    Disruptive Technology: Functional Languages Copyright© 2015 by Fred George. All rights reserved. 23 •Middleware exploiting: • Kafka written in Scala • RabbitMQ written in Erlang •Case study: MailOnline page rendering • Legacy system: 130K loc Java/JSP’s • Replacement: 4,000 loc Clojure
  • 23.
    Mitigate Process Inhibitors Copyright© 2015 by Fred George. All rights reserved. 24
  • 24.
    Understand Your Problem Copyright© 2015 by Fred George. All rights reserved. 25 Complex Simple Complicated Chaotic Cause a Effect Cause aaa Effect Effect? Cause? Cause ? Effect Disorder
  • 25.
    Copyright © 2015by Fred George. All rights reserved. 26 Requirements Trying ideas “Experimentation drives Innovation” New Vision of “What” to Build
  • 26.
    Copyright © 2015by Fred George. All rights reserved. 27 Task Story Feature Project Initiative Co-Credit: Greg Reiser, ThoughtWorks Re-think Interactions
  • 27.
    • Bad metrics: •Lines of code • Development stages and milestones met • Business success metrics: • Sales, clicks, registrations, customer retention, etc. Measure What Matters Copyright © 2015 by Fred George. All rights reserved. 28
  • 28.
    Mitigate Organization Inhibitors Copyright ©2015 by Fred George. All rights reserved. 29
  • 29.
    Over-Specialization Copyright © 2015by Fred George. All rights reserved. •Theory: • Specialist are more productive •Practice: • Overhead of communication is under-estimated • Unbalanced workload creates delays 30
  • 30.
    Specialization Institutionalized with Titles Copyright© 2015 by Fred George. All rights reserved. 31 Case Study: •50 IT professionals •25+ titles •0 people understanding projects
  • 31.
    Solution: Fix theTitles Copyright © 2015 by Fred George. All rights reserved. 32 Step 1: Define competence in key technologies Strategic Technologies • Ruby • Java • iOS • Android • DB (SQL and non-SQL) • Testing • OO Design • Node.js • HTML/CSS/JavaScript • …
  • 32.
    New Titles forNew Projects Copyright © 2015 by Fred George. All rights reserved. 33 Not competent in a key technology Competent in a key technology Expert in a key technology Competent in 5-7 key technologies Expert in 3-5 key technologies
  • 33.
    “Fix the Furniture”* Copyright© 2015 by Fred George. All rights reserved. 34*Kent Beck, XP Explained
  • 34.
    Non-dedicated Leadership Copyright ©2015 by Fred George. All rights reserved. 35
  • 35.
    Bring Work tothe Team… Copyright © 2015 by Fred George. All rights reserved. 36 Not musical chairs for each small project
  • 36.
    Copyright © 2015by Fred George. All rights reserved. 37 MicroServices Cloud Docker Dev/Ops Full-stack developer Agile Lean startup Cassandra Event bus MVP (minimum viable product Programmer anarchy No-SQL Managerless process
  • 37.
    Copyright © 2015by Fred George. All rights reserved. 38 MicroServices Cloud Docker Dev/Ops Full-stack developer Agile Lean startup Cassandra Event bus MVP (minimum viable product Programmer anarchy No-SQL Managerless process
  • 38.
    Project Delivery Cycles Copyright© 2015 by Fred George. All rights reserved. 39 NRK NRK 5 yrs. 1980 1990 2000 1 mon. 1 week 1 day 2010 1 year ProjectDuration(log) Waterfall Waterfall Agile Agile Waterfall w OO
  • 39.
    Acknowledgements Copyright © 2015by Fred George. All rights reserved. 40 •Neil Hutchinson – Founder of Forward • Adopting a “Fail Fast” culture •Carl Gaywood • Former programmer who wanted to Go Faster •Paul Ingles and Mike Jones • Implementers of “Experimentation drives innovation”
  • 40.
    More Information… •Google: • MicroServiceArchitecture for videos • MicroXchg 2015, 2016 conferences • MicroService track in most conferences • Programmer Anarchy for managerless teams • Docker for latest container deployment Copyright © 2015 by Fred George. All rights reserved. 41
  • 41.
    Go Faster: Tech, Process,& Organization Fred George fredgeorge@acm.org @fgeorge52 Copyright © 2015 by Fred George. All rights reserved. 42