This document discusses using Netflix's microservices stack on AWS. It describes Netflix's architecture of using hundreds of microservices across multiple regions to handle billions of requests per day. It outlines the principles of Netflix's stack including stateless services, auto-scaling, no single points of failure, and designing for failures. Key technologies in Netflix's open source stack are explained like Eureka for service discovery, Ribbon for load balancing, Hystrix for latency and fault tolerance, RxJava for reactive programming, and Dynomite for distributed caching. Chaos engineering practices like fault injection testing are also covered.

1. Microservices reativos usando
a stack do Netflix na AWS
Diego Pacheco
Principal Software Architect at ilegra.com
@diego_pacheco

3. www.ilegra.com

4. NetflixOSS
Stack

5. Why Netflix?
 Billions Requests Per Day
 1/3 US internet
bandwidth
 ~10k EC2 Instances
 Multi-Region
 100s Microservices
 Innovation + Solid
Service
 SOA, Microservices and
DevOps Benchmark
 Social Product
 Social Network
 Video
 Docs
 Apps
 Chat
Scalability
Distributed Teams
Could reach some
Web Scale
Netflix My Problem

6. AWS

7. Cloud Native

8. Principles
 Stateless Services
 Ephemeral Instances
 Everything fails all the
time
 Auto Scaling / Down
Scaling
 Multi AZ and multi
Region
 No SPOF
 Design for Failure
(expected)
 SOA
 Microservices
 No Central Database
 NoSQL
 Lightweight Serializable
Objects
 Latency tolerant
protocols
 DevOps Enabler

Immutable Infrastructure

Anti-Fragility

9. Right Set of
Assumptons

10. Microservices

11. Reactive

12. Java Drivers X REST
X

13. Simple View of the
Architecture
Zuul
UI
Microservice
Cassandra
Cluster

14. Stack

15. OSS

16. Zuul

17. Zuul

18. Karyon: Microbiology - Nucleus

19.  Reactive Extensions + Netty Server
 Lower Latency under Heavy Load
 Fewer Locks, Fewer Thread Migrations
 Consumes Less CPU
 Lower Object Allocation Rate
RxNetty

20. Karyon:
CODE

21. Karyon:
Reactive

22. Karyon:
Reactive

23. Eureka and Service
Discovery
http://microservices.io/patterns/server-side-discovery.html

24. Eureka
 AWS Service Registry for Mid-tier
 Load balancing and Failover
 REST based
 Karyon and Ribbon Integration

25. Eureka

26. Eureka and Service
Discovery

27. Availability

28. Histryx

29.  IPC Library
 Client Side Load Balancing
 Multi-Protocol (HTTP, TCP, UDP)
 Caching*
 Batching
 Reactive
Ribbon

30. Ribbon
CODE

31. Ribbon
CODE

32.  Reactive Extension of the JVM
 Async/Event based programming
 Observer Pattern
 Less 1mb
 Heavy usage by Netflix OSS Stack
RX-Java

33. Archaius
 Configuration Management Solution
 Dynamic and Typed Properties
 High Throughtput and Thread Safety
 Callbacks: Notifications of config changes
 JMX Beans
 Dynamic Config Sources: File, Db, DynamoDB, Zookeper
 Based on Apache Commons Configuration

34. Archaius + Git
MicroserviceMicroservice Slave Side Car
Central
Internal GIT
Property
Files
File
System
MicroserviceMicroservice Slave Side Car
File
System
MicroserviceMicroservice Slave Side Car
File
System

35. Asgard

36. Asgard

37. Packer
JOB
Create
Bake/Provision
Launch
Deploys

38. Dynomite: Distributed Cache
https://github.com/Netflix/dynomite

39. Dynomite
 Implements the Amazon Dynamo
Similar to Cassandra, Riak and DynamoDB
Strong Consistency – Quorum-like – No Data Loss
Pluggable
Scalable
Redis / Memcached
Multi-Clients with Dyno
Can use most of redis commands
Integrated with Eureka via Prana

40. Dynomite:
Internals

41. Oregon D1
Oregon D2
N California D3
Eureka Server
Eureka Server
Prana
Prana
Prana
Multi-Region
Cluster

42. Dynomite:
CODE

43. Dynomite
Contributions
https://github.com/Netflix/dynomite
https://github.com/Netflix/dynomite/pull/207
https://github.com/Netflix/dynomite/pull/200

44. Caos
Engineering

45.  Isolate Failure – Avoid cascading
 Redundancy – NO SPOF
 Auto-Scaling
 Fault Tolerance and Isolation
 Recovery
 Fallbacks and Degraded Experience
 Protect Customer from failures – Don’t throw Failures ->
Failures VS Errors
Chaos / Failure

46. Gatling
Stress Testing
Tool
Scala DSL
Run on top of Akka
Simple to use

47. Chaos
Arch
Zuul
Microservice N1 Microservice N2
Cassandra Cluster
Zuul
Eureka
ELB

48. Running…

49. Chaos Results and Learnings
 Retry configuration and Timeouts in Ribbon
 Right Class in Zuul 1.x (default retry only SocketException)

RequestSpecificRetryHandler (Httpclient Exceptions)

zuul.client.ribbon.MaxAutoRetries=1

zuul.client.ribbon.MaxAutoRetriesNextServer=1

zuul.client.ribbon.OkToRetryOnAllOperations=true
Eureka Timeouts
 It Works
 Everything needs to have redudancy
 ASG is your friend :-)
 Stateless Service FTW

50. Microservice Producer
Kafka / Storm :: Event System

51. Chaos Results and Learnings
Before:

Data was not in Elastic Search

Producers was loosing data

After:

No Data Loss

It Works

Changes:

No logging on Microservice :( (Log was added)

Code that publish events on a try-catch

Retry config in kafka producer from 0 to 5

53. Main
Challenges

54. Hacker
Mindset

56. Next Steps
 IPC
 Spinnaker
 Containers
 Client side Aggregation
 DevOps 2.0 -> Remediation / Skynet

57. Pocs
https://github.com/diegopacheco/netflixoss-pocs
http://diego-pacheco.blogspot.com.br/search/label/netflix?max-results=30

58. Microservices reativos usando
a stack do Netflix na AWS
Diego Pacheco
Principal Software Architect at ilegra.com
@diego_pacheco
Obrigado!