The document discusses Netflix's transition to a microservices architecture and its associated organizational changes following significant outages in 2008. It emphasizes the importance of a culture of full cycle developers, who are responsible for the entire lifecycle of the services they build, supported by specific tools and centralized teams. Additionally, it highlights the role of open-source software in facilitating these changes and improving operational resilience through practices like chaos engineering and continuous delivery.

1. Netflix MSA and Pivotal
Younjin Jeong
Principal Technologist
yjeong@pivotal.io

3. Agenda
■ Netflix Introduction
■ MSA backgrounds
■ Netflix’s expectations and
Considerations
■ Changes in Organizations
■ Netflix OSS for MSA
■ Pivotal
© Copyright 2018 Pivotal Software, Inc. All rights reserved.

4. Why Netflix Changed
Overview
© Copyright 2018 Pivotal Software, Inc. All rights reserved.

5. Why Netflix Changed
Netflix is a Video Streaming company that has 125M users in global, and 100k playing time per day.

6. Why Netflix Changed
Service in 190 counties in global, including South Korea since 2016

7. Why Netflix Changed
Netflix 2008 – Big Java Big Oracle
2008
2008, Netflix experienced huge outage.
It was started from Database Disk corruption, and whole service goes down.

8. Why Netflix Changed
Cloud IaaS
Data
Micro Services
Routing / Health check
API
Logging
Big Oracle
Big Java
Cloud Native Journey (2008 ~ )
It took many many years, and it’s still going on.

9. Why Netflix Changed
Netflix Today - Micro Services Architecture

10. Why Netflix Changed

11. Changes in Organizations
The Secret Ingredient of MSA
© Copyright 2018 Pivotal Software, Inc. All rights reserved.

12. Changes in Organizations
Netflix Considerations
Detailed Monitoring
Fault Isolation Control
API base Service interactions
Cloud Based Service Lifecycle Management
Fan-Out effect Control
Various Backing Services Support
Configurations Management for Number of Microservices
Service Auto Healing
Log Aggregation and Analysis
Fault Injection Test
Organizations
Culture
Configuration Server
Service Discovery
Circuit Breaker
API Gateway
Distributed Tracing
Zero Downtime Delivery
Fault Injection Test
Chaos Engineering
Persistence Cache Layer
Sidecar / Library
Freedom and Responsibility
Self-service Paradigm
Netflix Solutions
Org
Changes
Tech
Changes

13. You Build It, You Run It
You Support It
NETFLIX PRINCIPAL
© Copyright 2018 Pivotal Software, Inc. All rights reserved.
Netflix culture made all this possible.

14. Changes in Organizations
https://medium.com/netflix-techblog/full-cycle-developers-at-netflix-a08c31f83249
May 18, Netflix announced a concept of “Full Cycle Developers”

15. Changes in Organizations
Software Development Lifecycle, SDLC
Each specialist team creates tools for each cycles,
But the costs increased for communications, feedback loops, and bottlenecks.

16. Changes in Organizations
So they bring up an idea about SDLC + DevOps for “Operate what you Build”
Distributing this concept to every single development team, that they need to do
development issues, performance bugs, capacity planning, partner support, and so on.

17. Changes in Organizations
Netflix created centralized teams such as “Cloud Platform”, “Performance and Reliability E
ng.”, “Engineering Tools”
To support each development teams by “Tools” that can be easily re-used.
They have been created Common Netflix Tools and that was a key to scale their services.

18. Changes in Organizations
Netflix developers using this amazing tools, that is helping to be Full Cycle Developers.
Full Cycle Developers can thinks and act like SWE, SDET, and SRE.

19. Changes in Organizations
Trade-Offs of Full Cycle Developers
• Some developers wants to dive deep in narrow field. Netflix s
upport them to find right roles in the company. Others are enjoyin
g this Full cycle approach.
• Cloud Based systems were very helpful to achieve this concept, b
ut this broad approach increased developer load. So they rota
te on-calls for deployment + operations + support. It works when
works, but sometimes burnout everyone in the team.
• There’s no such a tool can solve every problem. They provide “pa
ve road” tools that created by specialists for each team, but
not limiting their people to use other tools. But those pave roa
d tools are very helpful and centralize, which means that specialis
t team’s efforts needed to aligning, and ongoing adoptions.

20. Changes in Organizations

21. Netflix OSS for MSA
Tools that made by Specialists
© Copyright 2018 Pivotal Software, Inc. All rights reserved.

22. Netflix OSS
Netflix Considerations
Detailed Monitoring
Fault Isolation Control
API base Service interactions
Cloud Based Service Lifecycle Management
Fan-Out effect Control
Various Backing Services Support
Configurations Management for Number of Microservices
Service Auto Healing
Log Aggregation and Analysis
Fault Injection Test
Organizations
Culture
Configuration Server
Service Discovery
Circuit Breaker
API Gateway
Distributed Tracing
Zero Downtime Delivery
Fault Injection Test
Chaos Engineering
Persistence Cache Layer
Sidecar / Library
Freedom and Responsibility
Self-service Paradigm
Netflix Solutions
Org
Changes
Tech
Changes

23. Netflix OSS
https://netflix.github.io/
• Externalized Config Server
• Service Register & Discovery
• Circuit Breaker
• API Gateway
• Load Balancing with IPC
• Realtime Monitoring
• Zero Downtime Delivery
• Fault Injection Test
• Chaos Engineering
• Etc…

24. Netflix OSS
Netflix Externalize Configuration – Archaius
https://medium.com/netflix-techblog/announcing-archaius-dynamic-properties-in-the-cloud-bc8c51faf675
Configuration Server1

25. Netflix OSS
Netflix Service Discovery - Eureka
http://techblog.netflix.com/2012/09/eureka.html
Service Discovery2

26. Netflix OSS
Netflix Circuit Breaker - Hystrix
https://medium.com/netflix-techblog/introducing-hystrix-for-resilience-engineering-13531c1ab362
Circuit Breaker – Fault Isolation3

27. Netflix OSS
Netflix API Gateway – Zuul
https://medium.com/netflix-techblog/announcing-zuul-edge-service-in-the-cloud-ab3af5be08ee
API Gateway4

28. Netflix OSS
Netflix Load Balancing with IPC – Ribbon
https://medium.com/netflix-techblog/announcing-ribbon-tying-the-netflix-mid-tier-services-together-a89346910a62
Load Balancing - Ribbon4

29. Netflix OSS
https://www.slideshare.net/brendangregg/how-netflix-tunes-ec2-instan
ces-for-performance
Netflix Realtime Monitoring – Atlas / Servo / Spectator
Distributed Tracing - Monitoring5

30. Netflix OSS
Distributed Tracing - Monitoring5

31. Netflix OSS
Netflix CI/CD – Spinnaker
https://medium.com/netflix-techblog/global-continuous-delivery-with-spinnaker-2a6896c23ba7
Zero Downtime Delivery - Deploy6

32. Netflix OSS
Netflix Zero Downtime Delivery – Kayenta
https://medium.com/netflix-techblog/automated-canary-analysis-at-netflix-with-kayenta-3260bc7acc69
Zero Downtime Delivery - Canary6

33. Netflix OSS
Netflix Zero Downtime Delivery – Kayenta
Zero Downtime Delivery6

34. Netflix OSS
Netflix Fault Injection Test – FIT
And Automate it
https://medium.com/netflix-techblog/fit-failure-injection-testing-35d8e2a9bb2
Fault Injection – Artificial Failure7

35. Netflix OSS
Netflix Chaos Engineering – Simain Army
https://medium.com/netflix-techblog/the-netflix-simian-army-16e57fbab116
Chaos Engineering8
Simian army tools are Killing Netflix Servers to verify what happens when its down.

36. Netflix OSS
Persistent Cache Layer9

37. Netflix OSS
Sidecar - Polyglot10

38. Built to Adapt
Overview
© Copyright 2018 Pivotal Software, Inc. All rights reserved.

39. Built to Adapt - Enterprise
You may heard about “Spring Cloud Netflix” to adopt Netflix like architectures.
Configuration Server1
Service Discovery2
Circuit Breaker – Fault Isolation3
API Gateway4
Load Balancing - Ribbon4

40. Built to Adapt - Enterprise
Distributed Tracing - Monitoring5

41. Built to Adapt - Enterprise
Distributed Tracing - Monitoring5

42. Built to Adapt - Enterprise
Zero Downtime Delivery6

43. Built to Adapt - Enterprise
Zero Downtime Delivery6

44. Built to Adapt - Enterprise
Persistent Cache Layer9
Spring Cache Abstrac
tion
Pivotal Cloud Cache
Pivotal Gemfire

45. Built to Adapt - Enterprise
All together10

46. Full Cycle Practice
Overview
© Copyright 2018 Pivotal Software, Inc. All rights reserved.

47. Full Cycle Practice with Pivotal
Netflix Considerations
Detailed Monitoring
Fault Isolation Control
API base Service interactions
Cloud Based Service Lifecycle Management
Fan-Out effect Control
Various Backing Services Support
Configurations Management for Number of Microservices
Service Auto Healing
Log Aggregation and Analysis
Fault Injection Test
Organizations
Culture
Configuration Server
Service Discovery
Circuit Breaker
API Gateway
Distributed Tracing
Zero Downtime Delivery
Fault Injection Test
Chaos Engineering
Persistence Cache Layer
Sidecar / Library
Freedom and Responsibility
Self-service Paradigm
Netflix Solutions
Org
Changes
Tech
Changes

48. Full Cycle Practice with Pivotal
�
☹
�
� �
� �
�
�
�
�

49. Full Cycle Practice with Pivotal
�
PCF Metrix
PivotalDataScience
PivotalBigDataSuite

50. Full Cycle Practice with Pivotal

51. Full Cycle Practice with Pivotal

52. Conclusion
Overview
© Copyright 2018 Pivotal Software, Inc. All rights reserved.

53. Conclusion
Spring eco system provides a “easy way” to build Micro Services and Cloud
Native Applications

54. Conclusion
Pivotal Cloud Foundry provides the “Platform” functionality to every single micro s
ervices.

55. Conclusion
Frameworks and Platform can help to achieve 12 factor application & operations,
Which means that can be used to help adopt Full Cycle Development teams.

56. Conclusion
Culture > Practice > Too
ls

57. Conclusion
If you want to build a ship,
don’t drum up the people
to gather wood, divide the
work, and give orders.
Instead, teach them to yearn
for the vast and endless sea.
by Antoine de Saint-Exupéry, the author of The Little Prince
from https://jobs.netflix.com/culture

58. Thank you
감사합니다