The document summarizes Adrian Cockcroft's experience giving talks about Netflix's approach to technology over time. It notes that initially people reacted skeptically, saying Netflix's approach was crazy and wouldn't work (2009-2010). Later, people said it could only work for large companies like Netflix (2011). By 2012, people said they wanted to adopt a similar approach but couldn't. The document outlines key lessons learned from Cockcroft's time at Netflix, including that speed wins in the marketplace and removing friction from product development helps enable faster innovation.

1. Fast Delivery
Adrian Cockcroft @adrianco
Technology Fellow - Battery Ventures
October 2014

2. Typical reactions to my Netflix talks…

3. Typical reactions to my Netflix talks…
“You guys are
crazy! Can’t
believe it”
– 2009

4. Typical reactions to my Netflix talks…
“You guys are
crazy! Can’t
believe it”
– 2009
“What Netflix is doing
won’t work”
– 2010

5. Typical reactions to my Netflix talks…
“You guys are
crazy! Can’t
believe it”
– 2009
“What Netflix is doing
won’t work”
– 2010 It only works for
‘Unicorns’ like
Netflix”
– 2011

6. Typical reactions to my Netflix talks…
“You guys are
crazy! Can’t
believe it”
– 2009
“What Netflix is doing
won’t work”
– 2010 It only works for
‘Unicorns’ like
Netflix”
– 2011
“We’d like to do
that but can’t”
– 2012

7. Typical reactions to my Netflix talks…
“You guys are
crazy! Can’t
believe it”
– 2009
“What Netflix is doing
won’t work”
– 2010 It only works for
‘Unicorns’ like
Netflix”
– 2011
“We’d like to do
that but can’t”
– 2012
“We’re on our way using
Netflix OSS code”
– 2013

8. What I learned from my time at Netflix

9. What I learned from my time at Netflix
•Speed wins in the marketplace

10. What I learned from my time at Netflix
•Speed wins in the marketplace
•Remove friction from product development

11. What I learned from my time at Netflix
•Speed wins in the marketplace
•Remove friction from product development
•High trust, low process, no hand-offs between teams

12. What I learned from my time at Netflix
•Speed wins in the marketplace
•Remove friction from product development
•High trust, low process, no hand-offs between teams
•Freedom and responsibility culture

13. What I learned from my time at Netflix
•Speed wins in the marketplace
•Remove friction from product development
•High trust, low process, no hand-offs between teams
•Freedom and responsibility culture
•Don’t do your own undifferentiated heavy lifting

14. What I learned from my time at Netflix
•Speed wins in the marketplace
•Remove friction from product development
•High trust, low process, no hand-offs between teams
•Freedom and responsibility culture
•Don’t do your own undifferentiated heavy lifting
•Use simple patterns automated by tooling

15. What I learned from my time at Netflix
•Speed wins in the marketplace
•Remove friction from product development
•High trust, low process, no hand-offs between teams
•Freedom and responsibility culture
•Don’t do your own undifferentiated heavy lifting
•Use simple patterns automated by tooling
•Self service cloud makes impossible things instant

16. Cloud Adoption
%*&!”
By Simon Wardley http://enterpriseitadoption.com/

17. Cloud Adoption
%*&!”
2009
By Simon Wardley http://enterpriseitadoption.com/

18. Cloud Adoption
%*&!”
2009
By Simon Wardley http://enterpriseitadoption.com/

19. Cloud Adoption
@adrianco’s
new job at the
intersection
of cloud and
Enterprise IT
2009 2014
%*&!”
By Simon Wardley http://enterpriseitadoption.com/

20. The Global Land-Grab

21. The Global Land-Grab

22. The Global Land-Grab

23. The Global Land-Grab
Azure
AWS
GCE
19 Regions
11 Regions
3 Regions

24. The Global Land-Grab
Azure
AWS
GCE
?
19 Regions
11 Regions
3 Regions
?
?
?
?
http://www.google.com/about/datacenters/inside/locations/index.html
http://azure.microsoft.com/en-us/regions/

25. The Global Land-Grab
Azure
AWS
GCE
?
19 Regions
11 Regions
3 Regions
?
?
?
?
http://www.google.com/about/datacenters/inside/locations/index.html
http://azure.microsoft.com/en-us/regions/

26. This is the year that Enterprises
finally embraced cloud.

27. This is the year that Enterprises
finally embraced cloud.

28. This is the year that Enterprises
finally embraced cloud.

29. This is the year that Enterprises
finally embraced cloud.

30. What separates
incumbents from
disruptors?

31. “It isn't what we don't know that
gives us trouble, it's what we
know that ain't so.”
!
Will Rogers

32. Assumptions

33. Optimizations

34. Assumption:
Process prevents
problems

35. Organizations build up
slow complex “Scar
tissue” processes

36. "This is the IT swamp draining manual for anyone who is
neck deep in alligators.”
1984 2014

37. Product
Development
Processes

38. Observe
Orient
Act Continuous
Delivery
Decide

39. Land grab
opportunity Competitive
Observe
Orient
Decide
Act
Move
Customer Pain
Point
Measure
Customers
Continuous
Delivery

40. Land grab
opportunity Competitive
Observe
Orient
Decide
Act
Move
Customer Pain
Point
INNOVATION
Measure
Customers
Continuous
Delivery

41. Land grab
opportunity Competitive
Observe
Orient
Decide
Act
Move
Customer Pain
Point
Analysis
Model
Hypotheses
INNOVATION
Measure
Customers
Continuous
Delivery

42. Land grab
opportunity Competitive
Observe
Orient
Decide
Act
Move
Customer Pain
Point
Analysis
BIG DATA
Model
Hypotheses
INNOVATION
Measure
Customers
Continuous
Delivery

43. Land grab
opportunity Competitive
Observe
Orient
Decide
Act
Move
Customer Pain
Point
Analysis
BIG DATA
Plan Response
JFDI
Share Plans
Model
Hypotheses
INNOVATION
Measure
Customers
Continuous
Delivery

44. Land grab
opportunity Competitive
Observe
Orient
Decide
Act
Move
Customer Pain
Point
Analysis
BIG DATA
Plan Response
JFDI
Share Plans
Model
Hypotheses
INNOVATION
CULTURE
Measure
Customers
Continuous
Delivery

45. Land grab
opportunity Competitive
Observe
Orient
Decide
Act
Move
Customer Pain
Point
Analysis
BIG DATA
Plan Response
JFDI
Share Plans
Launch AB
Test
Automatic
Deploy
Incremental
Features
Model
Hypotheses
INNOVATION
CULTURE
Measure
Customers
Continuous
Delivery

46. Land grab
opportunity Competitive
Observe
Orient
Decide
Measure
Customers
Act
Move
Customer Pain
Point
Analysis
BIG DATA
Plan Response
JFDI
Share Plans
Launch AB
Test
Automatic
Deploy
Incremental
Features
Model
Hypotheses
INNOVATION
CULTURE
CLOUD
Continuous
Delivery

47. Land grab
opportunity Competitive
Observe
Orient
Decide
Measure
Customers
Act
Move
Customer Pain
Point
Analysis
BIG DATA
Plan Response
JFDI
Share Plans
Launch AB
Test
Automatic
Deploy
Incremental
Features
Model
Hypotheses
INNOVATION
CULTURE
CLOUD
Continuous
Delivery

48. Land grab
opportunity Competitive
Observe
Orient
Decide
Measure
Customers
Act
Move
Customer Pain
Point
Analysis
BIG DATA
Plan Response
JFDI
Share Plans
Launch AB
Test
Automatic
Deploy
Incremental
Features
Model
Hypotheses
INNOVATION
CULTURE
CLOUD
Continuous
Delivery

49. Breaking Down the SILOs

50. Breaking Down the SILOs
Prod
Mgr
UX Dev QA DBA Sys
Adm Adm
Net
Adm
SAN

51. Breaking Down the SILOs
Prod
Mgr
UX Dev QA DBA Sys
Adm Adm
Net
Adm
SAN
Product Team Using Monolithic Delivery
Product Team Using Monolithic Delivery

52. Breaking Down the SILOs
Product Team Using Monolithic Delivery
Prod
UX Dev QA DBA Sys
Net
SAN
Mgr
Adm
Adm
Adm Product Team Using Microservices
Product Team Using Monolithic Delivery
Product Team Using Microservices
Product Team Using Microservices

53. Breaking Down the SILOs
Product Team Using Monolithic Delivery
Prod
UX Dev QA DBA Sys
Net
SAN
Mgr
Adm
Adm
Adm Product Team Using Microservices
Product Team Using Monolithic Delivery
Product Team Using Microservices Platform Team
Product Team Using Microservices

54. Breaking Down the SILOs
Product Team Using Monolithic Delivery
Prod
UX Dev QA DBA Sys
Net
SAN
Mgr
Adm
Adm
Adm Product Team Using Microservices
Product Team Using Monolithic Delivery
Platform Team A
P
I Product Team Using Microservices
Product Team Using Microservices

55. Breaking Down the SILOs
Product Team Using Monolithic Delivery
Prod
UX Dev QA DBA Sys
Net
SAN
Mgr
Adm
Adm
Adm Product Team Using Microservices
Product Team Using Monolithic Delivery
Platform Team
A
P
I Product Team Using Microservices
Product Team Using Microservices
DevOps is a Re-Org

56. Release Plan
Developer
Developer
Developer
Developer
Developer
QA Release
Integration
Ops Replace Old
With New
Release
Monolithic service updates
Works well with a small number
of developers and a single
language like php, java or ruby

57. Release Plan
Developer
Developer
Developer
Developer
Developer
Monolithic service updates
QA Release
Integration
Ops Replace Old
With New
Release
Bugs
Works well with a small number
of developers and a single
language like php, java or ruby

58. Release Plan
Developer
Developer
Developer
Developer
Developer
Monolithic service updates
QA Release
Integration
Ops Replace Old
With New
Release
Bugs
Bugs
Works well with a small number
of developers and a single
language like php, java or ruby

59. Developer
Developer
Developer
Developer
Developer
Old Release Still
Running
Release Plan
Release Plan
Release Plan
Release Plan
Immutable microservice deployment
is faster, scales with large teams and
diverse platform components

60. Developer
Developer
Developer
Developer
Developer
Immutable microservice deployment
is faster, scales with large teams and
diverse platform components
Old Release Still
Running
Release Plan
Release Plan
Release Plan
Release Plan
Deploy
Feature to
Production
Deploy
Feature to
Production
Deploy
Feature to
Production
Deploy
Feature to
Production

61. Developer
Developer
Developer
Developer
Developer
Immutable microservice deployment
is faster, scales with large teams and
diverse platform components
Old Release Still
Running
Release Plan
Release Plan
Release Plan
Release Plan
Deploy
Feature to
Production
Deploy
Feature to
Production
Deploy
Feature to
Production
Deploy
Feature to
Production
Bugs

62. Developer
Developer
Developer
Developer
Developer
Immutable microservice deployment
is faster, scales with large teams and
diverse platform components
Old Release Still
Running
Release Plan
Release Plan
Release Plan
Release Plan
Deploy
Feature to
Production
Deploy
Feature to
Production
Deploy
Feature to
Production
Deploy
Feature to
Production
Bugs
Deploy
Feature to
Production

63. Developing at the Speed of Docker
Developers
• Compile/Build
• Seconds
Extend container
• Package dependencies
• Seconds
PaaS deploy Container
• Docker startup
• Seconds
etc…

64. Developing at the Speed of Docker
Developers
• Compile/Build
• Seconds
Extend container
• Package dependencies
• Seconds
PaaS deploy Container
• Docker startup
• Seconds
Emerging market for Docker runtime orchestration options
etc…

65. What Happened?
Rate of change
increased
Cost and size and
risk of change
reduced

66. Disruptor:
Continuous Delivery
with Microservices

67. A Microservice Definition
!
Loosely coupled service oriented
architecture with bounded contexts

68. If every service has to be
updated at the same time
it’s not loosely coupled
A Microservice Definition
!
Loosely coupled service oriented
architecture with bounded contexts

69. If every service has to be
updated at the same time
it’s not loosely coupled
A Microservice Definition
!
Loosely coupled service oriented
architecture with bounded contexts
If you have to know too much about surrounding
services you don’t have a bounded context. See the
Domain Driven Design book by Eric Evans.

70. High Availability Patterns
● Business logic isolation in stateless micro-services
● Immutable code with instant rollback
● Auto-scaled capacity and deployment updates
● Distributed across availability zones and regions
● De-normalized single function NoSQL data stores
● See over 40 NetflixOSS projects at netflix.github.com
● Get “Technical Indigestion” trying to keep up with techblog.netflix.com

71. US Bandwidth April 2014

72. US Bandwidth April 2014
ELB

73. US Bandwidth April 2014
ELB
OpenConnect

74. Cloud Native
Monitoring and
Microservices

75. Cloud Native
● High rate of change
Code pushes can cause floods of new instances and metrics
Short baseline for alert threshold analysis – everything looks unusual
● Ephemeral Configurations
Short lifetimes make it hard to aggregate historical views
Hand tweaked monitoring tools take too much work to keep running
● Microservices with complex calling patterns
End-to-end request flow measurements are very important
Request flow visualizations get overwhelmed

76. Microservice Based Architectures
See http://www.slideshare.net/LappleApple/gilt-from-monolith-ruby-app-to-micro-service-scala-service-architecture

77. “Death Star” Architecture Diagrams
As visualized by Appdynamics, Boundary.com and Twitter internal tools

78. “Death Star” Architecture Diagrams
Netflix Gilt Groupe (12 of 450) Twitter
As visualized by Appdynamics, Boundary.com and Twitter internal tools

79. Continuous Delivery and DevOps
● Changes are smaller but more frequent
● Individual changes are more likely to be broken
● Changes are normally deployed by developers
● Feature flags are used to enable new code
● Instant detection and rollback matters much more

80. Whoops! I didn’t mean that!
Reverting…
Not cool if it takes 5 minutes to see it failed and 5 more to see a fix
No-one notices if it only takes 5 seconds to detect and 5 to see a fix

81. NetflixOSS Hystrix/Turbine Circuit Breaker
http://techblog.netflix.com/2012/12/hystrix-dashboard-and-turbine.html

82. NetflixOSS Hystrix/Turbine Circuit Breaker
http://techblog.netflix.com/2012/12/hystrix-dashboard-and-turbine.html

83. Low Latency SaaS Based Monitors
www.vividcortex.com and www.boundary.com

84. Metric to display latency needs to be
less than human attention span (~10s)

85. Separation of Concerns
Bounded Contexts

86. Answers to Some
Common Questions

87. Reaction from Fortune 100 CTO:
“But Netflix has a superstar development team, we don’t!"

88. Reaction from Fortune 100 CTO:
“But Netflix has a superstar development team, we don’t!"
Adrian’s Response:
“Netflix hired them from you, and got out of their way…”

89. Where to Start with DevOps?
Enterprise Mobile Apps
Horizontal Team
App-Store Provisioning
APIs to Everyone
DevOps Already…

90. 1
Where to Start with Cloud?
Archive
Data in
the Cloud
2
Use Diverse
Cloud Zones
and Regions
3
Migrate to
Cloud-aware
Tooling

91. What about cloud
costs?

92. See my AWS Re:Invent talk
!
“Cloud Native Cost
Optimization”

93. Cloud Native Cost Optimization
Turn it off!
Capacity on demand
Consolidate and Reserve
Plan for price cuts
FOSS tooling
$$ $

94. Forward Thinking

95. Forward Thinking

96. Forward Thinking

97. Forward Thinking
http://eugenedvorkin.com/seven-micro-services-architecture-advantages/

98. Any Questions?
● Battery Ventures http://www.battery.com
● Adrian’s Blog http://perfcap.blogspot.com
● Slideshare http://slideshare.com/adriancockcroft
!
● Monitorama Opening Keynote Portland OR - May 7th, 2014 - Video available
● GOTO Chicago Opening Keynote May 20th, 2014 - Video available
● Qcon New York – Speed and Scale - June 11th, 2014 - Video available
● Structure - Cloud Trends - San Francisco - June 19th, 2014 - Video available
● GOTO Copenhagen/Aarhus – Denmark – Sept 25th, 2014
● DevOps Enterprise Summit - San Francisco - Oct 21-23rd, 2014
● GOTO Berlin - Germany - Nov 6th, 2014
● AWS Re:Invent - Las Vegas - November 14th, 2014
● Dockercon Europe - Amsterdam - December 4th, 2014
Disclosure: some of the companies mentioned are Battery Ventures Portfolio Companies
See www.battery.com for a list of portfolio investments