Slides for our O'Reilly Open Source Software Conference "Break me if you can: practical guide to building fault-tolerant systems" talk: https://conferences.oreilly.com/oscon/oscon-or/public/schedule/detail/75311

1. Break Me If You Can
Practical Guide to Building Fault-tolerant Systems
O'Reilly Open Source Conference, Portland OR,
July 18, 2019
Alex Borysov, Software Engineer @ Netflix
Mykyta Protsenko, Software Engineer @ Netflix

2. Who are we?
Alex Borysov
Software Engineer @Netflix
Mykyta Protsenko
Software Engineer @Netflix
@aiborisov
@mykyta_p
@WeAreNetflix

3. Fault-Tolerance?
@aiborisov
@mykyta_p

4. Fault vs Error vs Failure
@aiborisov
@mykyta_p

5. @aiborisov
@mykyta_p
Fault
@aiborisov
@mykyta_p
incorrect
internal
state
Picture by Bob McMillan. Public domain. See slide #135 for details.

6. @aiborisov
@mykyta_p
Error
@aiborisov
@mykyta_p
visibly
incorrect
behaviour
Picture by David Goehring. CC BY 2.0. See slide #135 for details.

7. @aiborisov
@mykyta_p
Failure
@aiborisov
@mykyta_p
main
functionality
is broken
Picture by Camerafiend. CC BY-SA 3.0. See slide #135 for details.

8. @aiborisov
@mykyta_p
RMS Titanic vs Miracle on the Hudson
@aiborisov
@mykyta_p
Willy Stöwer. Public domain. See slide #135 for details. By Greg Lam Pak Ng. CC BY 2.0. See slide #136 for details.

9. @aiborisov
@mykyta_p
RMS Titanic
@aiborisov
@mykyta_p
Fault: Hitting an iceberg
Error: Water in the hull
Failure: Sinking
Willy Stöwer. Public domain. See slide #135 for details.

10. @aiborisov
@mykyta_p
Miracle on the Hudson
@aiborisov
@mykyta_p
Fault: Hitting geese at 2818 ft
Error: Engines shut down
No Failure!
By Greg Lam Pak Ng. CC BY 2.0. See slide #136 for details.

11. Fault Error Failure
@aiborisov
@mykyta_p
→ →

12. Fault Error Failure
@aiborisov
@mykyta_p
→ →

13. @aiborisov
@mykyta_p
Fault Tolerance
@aiborisov
@mykyta_p
Code and Design Patterns
Product-Driven Decisions
Communication
By Greg Lam Pak Ng. CC BY 2.0. See slide #136 for details.

14. Dodging Geese
@aiborisov
@mykyta_p

15. @aiborisov
@mykyta_p
Dodging Geese Architecture
TOP-5
Geese Service
Clouds Service
Leaderboard Service
API
Gateway
@aiborisov
@mykyta_p
See slides ##135, 136 for licensing details.

16. @aiborisov
@mykyta_p
Dodging Geese Architecture
TOP-5
Geese Service
Clouds Service
Leaderboard Service
API
Gateway
@aiborisov
@mykyta_p

17. @aiborisov
@mykyta_p
Dodging Geese Architecture
TOP-5
Geese Service
Leaderboard Service
API
Gateway
@aiborisov
@mykyta_p
Clouds Service

18. @aiborisov
@mykyta_p
Dodging Geese Architecture
TOP-5
Leaderboard Service
API
Gateway
@aiborisov
@mykyta_p
Clouds Service
Geese Service

19. @aiborisov
@mykyta_p
Dodging Geese Architecture
Geese Service
Clouds ServiceAPI
Gateway
@aiborisov
@mykyta_p
TOP-5
Leaderboard Service

20. @aiborisov
@mykyta_p
Dodging Geese Architecture
TOP-5
Geese Service
Clouds Service
Leaderboard Service
API
Gateway
@aiborisov
@mykyta_p

21. @aiborisov
@mykyta_p
Dodging Geese Architecture
TOP-5
Geese Service
Clouds Service
Leaderboard Service
API
Gateway
@aiborisov
@mykyta_p

22. @aiborisov
@mykyta_p
Dodging Geese Architecture
TOP-5
Geese Service
Clouds Service
Leaderboard Service
API
Gateway
@aiborisov
@mykyta_p

23. @aiborisov
@mykyta_p
Leaderboard API (REST)
/players/<username>/score
{"name": "Jane", "score": 100}
/leaderboard/top/<n>
[{"name": "Jane", "score": 100},
{"name": "John", "score": 50}, ...]
@aiborisov
@mykyta_p

24. @aiborisov
@mykyta_p
gRPC Service Definitions
@aiborisov
@mykyta_p
service GeeseService {
// Return next line of geese.
rpc GetGeese (GetGeeseRequest) returns (GeeseResponse);
}

25. @aiborisov
@mykyta_p
gRPC Service Definitions
@aiborisov
@mykyta_p
service GeeseService {
// Return next line of geese.
rpc GetGeese (GetGeeseRequest) returns (GeeseResponse);
}
service CloudsService {
// Return next line of clouds.
rpc GetClouds (GetCloudsRequest) returns (CloudsResponse);
}

26. @aiborisov
@mykyta_p
service FixtureService {
// Return next line of geese and clouds.
rpc GetFixture (GetFixtureRequest) returns (FixtureResponse);
}
gRPC Gateway Service
@aiborisov
@mykyta_p

27. @aiborisov
@mykyta_p
service FixtureService {
// Return next line of geese and clouds.
rpc GetFixture (GetFixtureRequest) returns (FixtureResponse);
}
+ = Fixture
gRPC Gateway Service
@aiborisov
@mykyta_p

28. @aiborisov
@mykyta_p
public class FixtureService extends FixtureServiceImplBase {
Gateway Fixture Service
@aiborisov
@mykyta_p

29. @aiborisov
@mykyta_p
Gateway Fixture Service
API
Gateway
@aiborisov
@mykyta_p
Geese Service
Clouds Service

30. @aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Non-Blocking Calls
Don’t block
Send requests in parallel
Combine results when ready

31. @aiborisov
@mykyta_p
public class FixtureService extends FixtureServiceImplBase {
Gateway Service Implementation
@aiborisov
@mykyta_p
private final GeeseServiceFutureStub geeseClient = ...;
private final CloudsServiceFutureStub cloudsClient = ...;

32. @aiborisov
@mykyta_p
public class FixtureService extends FixtureServiceImplBase {
Gateway Service Implementation
@aiborisov
@mykyta_p
private final GeeseServiceFutureStub geeseClient = ...;
private final CloudsServiceFutureStub cloudsClient = ...;
@Override
public void getFixture(GetFixtureRequest request, StreamObserver<FixtureResponse> response) {
ListenableFuture<GeeseResponse> geese = geeseClient.getGeese(toGeese(request));
ListenableFuture<CloudsResponse> clouds = cloudsClient.getClouds(toClouds(request));
ListenableFuture<List<GeneratedMessageV3>> geeseAndClouds =
Futures.allAsList(geese, clouds);
...

33. @aiborisov
@mykyta_p
public class FixtureService extends FixtureServiceImplBase {
Gateway Service Implementation
@aiborisov
@mykyta_p
private final GeeseServiceFutureStub geeseClient = ...;
private final CloudsServiceFutureStub cloudsClient = ...;
@Override
public void getFixture(GetFixtureRequest request, StreamObserver<FixtureResponse> response) {
ListenableFuture<GeeseResponse> geese = geeseClient.getGeese(toGeese(request));
ListenableFuture<CloudsResponse> clouds = cloudsClient.getClouds(toClouds(request));
ListenableFuture<List<GeneratedMessageV3>> geeseAndClouds =
Futures.allAsList(geese, clouds);
...

34. @aiborisov
@mykyta_p

35. @aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Slow dependencies
Slow upstream services

36. @aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Timeouts
Guaranteed latency
for integration points

37. @aiborisov
@mykyta_p
public class FixtureService extends FixtureServiceImplBase {
...
Gateway Service Implementation
@aiborisov
@mykyta_p
@Override
public void getFixture(GetFixtureRequest request, StreamObserver<FixtureResponse> response) {
ListenableFuture<GeeseResponse> geese =
geeseClient.getGeese(toGeese(request));
ListenableFuture<CloudsResponse> clouds =
cloudsClient.getClouds(toClouds(request));
ListenableFuture<List<GeneratedMessageV3>> geeseAndClouds =
Futures.allAsList(geese, clouds);
...

38. @aiborisov
@mykyta_p
public class FixtureService extends FixtureServiceImplBase {
...
Gateway Service Implementation
@aiborisov
@mykyta_p
@Override
public void getFixture(GetFixtureRequest request, StreamObserver<FixtureResponse> response) {
ListenableFuture<GeeseResponse> geese =
geeseClient.withDeadlineAfter(500, MILLISECONDS).getGeese(toGeeseRequest(request));
ListenableFuture<CloudsResponse> clouds =
cloudsClient.withDeadlineAfter(500, MILLISECONDS).getClouds(toCloudsRequest(request));
ListenableFuture<List<GeneratedMessageV3>> geeseAndClouds =
Futures.allAsList(geese, clouds);
...

39. @aiborisov
@mykyta_p
@Override
public void getFixture(GetFixtureRequest request, StreamObserver<FixtureResponse> response) {
ListenableFuture<GeeseResponse> geese =
geeseClient.withDeadlineAfter(500, MILLISECONDS).getGeese(toGeeseRequest(request));
ListenableFuture<CloudsResponse> clouds =
cloudsClient.withDeadlineAfter(500, MILLISECONDS).getClouds(toCloudsRequest(request));
ListenableFuture<List<GeneratedMessageV3>> geeseAndClouds =
Futures.allAsList(geese, clouds);
...
public class FixtureService extends FixtureServiceImplBase {
...
Gateway Service Implementation
@aiborisov
@mykyta_p

40. @aiborisov
@mykyta_p
REST: Non-Blocking Calls
CompletableFuture<List<LeaderboardEntry>> leaderboard =
httpClient
.get().uri("/top/5")
.exchange()
.timeout(Duration.ofMillis(500))
.flatMap(cr -> cr.bodyToMono(...))
.toFuture();
@aiborisov
@mykyta_p

41. @aiborisov
@mykyta_p
REST: Non-Blocking Calls with Timeout
CompletableFuture<List<LeaderboardEntry>> leaderboard =
httpClient
.get().uri("/top/5")
.exchange()
.timeout(Duration.ofMillis(500))
.flatMap(cr -> cr.bodyToMono(...))
.toFuture();
@aiborisov
@mykyta_p

42. @aiborisov
@mykyta_p

43. Demo
@aiborisov
@mykyta_p

44. @aiborisov
@mykyta_p
@aiborisov
@mykyta_p
No Geese
No Clouds
Blinking Leaderboard

45. @aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Observability
Monitoring:
QPS, latency, errors, ...

46. @aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Observability: gRPC
Monitoring:
QPS, latency, errors, ...
// OpenCensus
RpcViews.registerAllViews();

47. @aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Tracing: gRPC
GrpcTracing grpcTracing =
GrpcTracing.create(...);
ManagedChannelBuilder
...
.intercept(grpcTracing.newClientInterceptor())
.build() ;
ServerBuilder.forPort(8080)
...
.intercept(grpcTracing.newServerInterceptor())
.build();

48. @aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Tracing: gRPC
GrpcTracing grpcTracing =
GrpcTracing.create(...);
ManagedChannelBuilder
...
.intercept(grpcTracing.newClientInterceptor())
.build();
ServerBuilder.forPort(8080)
...
.intercept(grpcTracing.newServerInterceptor())
.build();

49. @aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Tracing: REST
build.gradle:
dependencies {
compile '...:spring-cloud-sleuth-zipkin'
compile '...:spring-cloud-starter-sleuth'
...
}
application.properties:
spring.zipkin.baseUrl=http://zipkin:9411/
spring.sleuth.sampler.probability=1.0
spring.sleuth.web.enabled=true

50. @aiborisov
@mykyta_p

51. Demo
@aiborisov
@mykyta_p

52. @aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Clouds are slow
Geese are fast
Entire call fails

53. @aiborisov
@mykyta_p
ListenableFuture<GeeseResponse> geese =
geeseClient..getGeese(toGeese(request));
ListenableFuture<CloudsResponse> clouds =
cloudsClient.getClouds(toClouds(request));
ListenableFuture<List<GeneratedMessageV3>>
geeseAndClouds =
Futures.allAsList(geese, clouds);
...
@aiborisov
@mykyta_p
Partial Degradation

54. @aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Partial Degradation
ListenableFuture<GeeseResponse> geese =
geeseClient..getGeese(toGeese(request));
ListenableFuture<CloudsResponse> clouds =
cloudsClient.getClouds(toClouds(request));
ListenableFuture<List<GeneratedMessageV3>>
geeseAndClouds =
Futures.successfulAsList(geese, clouds);
...

55. @aiborisov
@mykyta_p

56. @aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Some L-board calls fail
L-board latency is low
Scores disappear

57. @aiborisov
@mykyta_p
CompletableFuture<List<Leaderboard>> request() {
return httpClient
.get().uri("/top/5").exchange()
.timeout(Duration.ofMillis(500))
.flatMap(...).toFuture();
}
@aiborisov
@mykyta_p
Retries: REST

58. @aiborisov
@mykyta_p
CompletableFuture<List<Leaderboard>> request() {
return httpClient
.get().uri("/top/5").exchange()
.timeout(Duration.ofMillis(500))
.flatMap(...).toFuture();
}
RetryPolicy RETRY_POLICY = new RetryPolicy()
.retryOn(IOException.class)
.withMaxRetries(MAX_RETRIES);
CompletableFuture<List<Leaderboard>> top5 =
Failsafe.with(RETRY_POLICY)
...
.future(this::httpRequest);
@aiborisov
@mykyta_p
Retries: REST

59. @aiborisov
@mykyta_p

60. Demo
@aiborisov
@mykyta_p

61. @aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Retry slow calls?
Retry failed calls?
Retry network faults?

62. @aiborisov
@mykyta_p
Retry Storm
Clouds ServiceAPI
Gateway
@aiborisov
@mykyta_p

63. @aiborisov
@mykyta_p
new RetryPolicy()
.withBackoff(
MIN_DELAY,
MAX_DELAY,
TimeUnit.MILLISECONDS, 100.0)
...
...
@aiborisov
@mykyta_p
Exponential Backoffs

64. @aiborisov
@mykyta_p
CircuitBreaker CIRCUIT_BREAKER =
new CircuitBreaker()
.withFailureThreshold(...);
Failsafe
.with(CIRCUIT_BREAKER)
.withFallback(
() -> emptyLeaderboard())
...
@aiborisov
@mykyta_p
Fallbacks

65. @aiborisov
@mykyta_p
CircuitBreaker CIRCUIT_BREAKER =
new CircuitBreaker()
.withFailureThreshold(...);
Failsafe
.with(CIRCUIT_BREAKER)
() -> cachedLeaderboard())
...
@aiborisov
@mykyta_p
Fallbacks

66. @aiborisov
@mykyta_p
Retry
Fallback
Fail Fast
@aiborisov
@mykyta_p
On Error

67. @aiborisov
@mykyta_p

68. @aiborisov
@mykyta_p
@aiborisov
@mykyta_p

69. @aiborisov
@mykyta_p
@aiborisov
@mykyta_p
High 99%ile latency
100 requests
Error probability?

70. @aiborisov
@mykyta_p
@aiborisov
@mykyta_p
High 99%ile latency
100 requests
Error probability:
1 – 0.99^100 = 63%

71. @aiborisov
@mykyta_p
Tail-Tolerance
@aiborisov
@mykyta_p
Request
200 ms deadline

72. @aiborisov
@mykyta_p
Tail-Tolerance
@aiborisov
@mykyta_p
Request
200 ms deadline
↓ 100 ms

73. @aiborisov
@mykyta_p
Tail-Tolerance
@aiborisov
@mykyta_p
Request
200 ms deadline
↓ 100 ms
Request

74. @aiborisov
@mykyta_p
Tail-Tolerance
@aiborisov
@mykyta_p
Request
200 ms deadline
↓ 100 ms
Request
Fastest Response

75. @aiborisov
@mykyta_p
High 99%ile latency
100 requests
@aiborisov
@mykyta_p
Request Hedging

76. @aiborisov
@mykyta_p
High 99%ile latency
100 requests
Error probability:
63% x 0.01 < 1%
@aiborisov
@mykyta_p
Request Hedging

77. @aiborisov
@mykyta_p
Channel geeseChannel = ManagedChannelBuilder
.forAddress(geeseHost, geesePort)
.enableRetry()
.maxHedgedAttempts(MAX_HEDGES)
.build();
GeeseServiceFutureStub geeseStub =
GeeseServiceGrpc
.newFutureStub(geeseChannel);
@aiborisov
@mykyta_p
Hedging in gRPC (soon)

78. @aiborisov
@mykyta_p
Channel geeseChannel = ManagedChannelBuilder
.forAddress(geeseHost, geesePort)
.enableRetry()
.maxHedgedAttempts(MAX_HEDGES)
.build();
GeeseServiceFutureStub geeseStub =
GeeseServiceGrpc
.newFutureStub(geeseChannel);
@aiborisov
@mykyta_p
Hedging in gRPC (soon)

79. @aiborisov
@mykyta_p

80. @aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Error Handling
Consistent Faults Fail Fast
Intermittent Slow Hedging
Intermittent Fast Retry
Fallback✚

81. @aiborisov
@mykyta_p

82. @aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Client-driven deadline
Don’t process failed calls

83. @aiborisov
@mykyta_p
Deadlines
API
Gateway
@aiborisov
@mykyta_p
See slides ##135, 136 for licensing details.

84. @aiborisov
@mykyta_p
Deadlines
API
Gateway
@aiborisov
@mykyta_p
Deadline 200 ms
→

85. @aiborisov
@mykyta_p
Deadlines
API
Gateway
@aiborisov
@mykyta_p
Deadline 200 ms
→ Spent 120 ms
→

86. @aiborisov
@mykyta_p
Deadlines
API
Gateway
@aiborisov
@mykyta_p
Spent 120 ms
→ Spent 90 ms
Deadline 200 ms
→
X

87. @aiborisov
@mykyta_p
Deadlines
API
Gateway
@aiborisov
@mykyta_p
Spent 120 ms
→ Spent 90 ms
Deadline 200 ms
→
X
→

88. @aiborisov
@mykyta_p
Deadline Propagation
API
Gateway
@aiborisov
@mykyta_p
Deadline 200 ms
→

89. @aiborisov
@mykyta_p
Deadline 80 ms
Deadline Propagation
API
Gateway
@aiborisov
@mykyta_p
Deadline 200 ms
→ Spent 120 ms
→

90. @aiborisov
@mykyta_p
Deadline 80 ms
Deadline Propagation
API
Gateway
@aiborisov
@mykyta_p
Spent 120 ms
→ Spent 90 ms
Deadline 200 ms
→
X

91. @aiborisov
@mykyta_p
Deadline 80 ms
Deadline Propagation
API
Gateway
@aiborisov
@mykyta_p
Spent 120 ms
→ Spent 90 ms
Deadline -10 ms
Deadline 200 ms
→
X

92. @aiborisov
@mykyta_p

93. @aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Throughput has limits
Exceeding limits?

94. @aiborisov
@mykyta_p
new ConcurrencyLimitServletFilter(
new ServletLimiterBuilder()
.partitionByHeader("GEESE_TYPE",
c -> c.assign("premium", 0.9)
.assign("free", 0.1))
.limiter(l -> l.limit(
newBuilder()
.initialLimit(1000)...);
@aiborisov
@mykyta_p
REST

95. @aiborisov
@mykyta_p
new ConcurrencyLimitServletFilter(
new ServletLimiterBuilder()
.partitionByHeader("GEESE_TYPE",
c -> c.assign("premium", 0.9)
.assign("free", 0.1))
.limiter(l -> l.limit(
newBuilder()
.initialLimit(1000)...);
@aiborisov
@mykyta_p
REST

96. @aiborisov
@mykyta_p
var limiter =
new GrpcServerLimiterBuilder()
.partitionByHeader(GEESE_TYPE)
.partition("premium", 0.9)
.partition("free", 0.1)
.limiter(l ->
l.limit(
newBuilder()
.initialLimit(1000)...);
ConcurrencyLimitServerInterceptor
.newBuilder(limiter).build();
@aiborisov
@mykyta_p
gRPC: Server

97. @aiborisov
@mykyta_p
var limiter =
new GrpcServerLimiterBuilder()
.partitionByHeader(GEESE_TYPE)
.partition("premium", 0.9)
.partition("free", 0.1)
.limiter(l ->
l.limit(
newBuilder()
.initialLimit(1000)...);
ConcurrencyLimitServerInterceptor
.newBuilder(limiter).build();
@aiborisov
@mykyta_p
gRPC: Server

98. @aiborisov
@mykyta_p
new GrpcClientLimiterBuilder()
.limit(
newBuilder()
.initialLimit(1000).build())
.blockOnLimit(false) // fail-fast
.build();
@aiborisov
@mykyta_p
gRPC: Client

99. @aiborisov
@mykyta_p

100. Demo
@aiborisov
@mykyta_p

101. Demo
@aiborisov
@mykyta_p

102. @aiborisov
@mykyta_p
Monitoring
@aiborisov
@mykyta_p
APM
Service
metrics
Distributed
tracing
Business
metrics
Picture by Alex Borysov. CC BY 2.0. See slide #136 for details.

103. @aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Code and Design
Timeouts / Deadline Propagation
Retries / Hedging
Proper Fallbacks
Concurrency Limits
Load Shedding
Observability

104. Demo
@aiborisov
@mykyta_p

105. @aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Bad user experience
Metrics are not enough

106. @aiborisov
@mykyta_p
Prober
TOP-5
API
Gateway
@aiborisov
@mykyta_p

107. @aiborisov
@mykyta_p
Prober
TOP-5
API
Gateway
@aiborisov
@mykyta_p
See slides ##135, 137 for licensing details.

108. @aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Prober
Availability
Latency SLO
Response verification

109. @aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Prober
Availability
Latency SLO
Response verification
CloudProber.org

110. @aiborisov
@mykyta_p

111. @aiborisov
@mykyta_p

112. @aiborisov
@mykyta_p

113. @aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Technical
solutions
are not enough

114. @aiborisov
@mykyta_p
Communication
@aiborisov
@mykyta_p

115. @aiborisov
@mykyta_p
Communication
@aiborisov
@mykyta_p

116. @aiborisov
@mykyta_p
Communication Channels
@aiborisov
@mykyta_p
GEESE
at 270

117. @aiborisov
@mykyta_p
Communication Channels
@aiborisov
@mykyta_p
GEESE
at 270

118. @aiborisov
@mykyta_p
GEESE
at 270
Communication Channels
@aiborisov
@mykyta_p

119. @aiborisov
@mykyta_p
GEESE
at 270
Communication Channels
@aiborisov
@mykyta_p

120. @aiborisov
@mykyta_p
Postmortems
@aiborisov
@mykyta_p
Blameless
Constructive

121. @aiborisov
@mykyta_p
Postmortems
@aiborisov
@mykyta_p
Blameless
Constructive
Social
See slides ##135, 137 for licensing details.

122. @aiborisov
@mykyta_p
Postmortems
@aiborisov
@mykyta_p
Timeline
Causes
Remedies
See slides ##135, 137 for licensing details.

123. @aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Learn from Failure
Blameless postmortems
Alert playbooks
Incident knowledge base

124. @aiborisov
@mykyta_p

125. @aiborisov
@mykyta_p
Libraries and Tools
@aiborisov
@mykyta_p
Demo: github.com/break-me-if-you-can
Failsafe: github.com/jhalterman/failsafe
Observability: opencensus.io + opentracing.io = opentelemetry.io
Prober: cloudprober.org
Concurrency Limits: github.com/Netflix/concurrency-limits

126. @aiborisov
@mykyta_p
Demo UI
@HalloGene_
Yevgen Golubenko
Twitter: @HalloGene_
github.com/HalloGene
Picture by Yevgen Golubenko. Also see slide #138 for licensing details.

127. @aiborisov
@mykyta_p
Books
@aiborisov
@mykyta_p

128. @aiborisov
@mykyta_p
@aiborisov
@mykyta_p
Fault-Tolerance
Code & Design Patterns
Product decisions
Communication culture

129. @aiborisov
@mykyta_p

130. @aiborisov
@mykyta_p
Please Rate Our Talk
Conference Website O’Reilly Events App

131. @aiborisov
@mykyta_p
Please Rate Our Talk
Conference Website O’Reilly Events App
See slides ##135, 137 for licensing details.

132. @aiborisov
@mykyta_p
Please Rate Our Talk
Conference Website O’Reilly Events App
See slides ##135, 137 for licensing details.

133. @aiborisov
@mykyta_p
Please Rate Our Talk
Conference Website O’Reilly Events App
5 STARS!
See slides ##135, 137 for licensing details.

134. @aiborisov
@mykyta_p

135. @aiborisov
@mykyta_p
Images and Licensing
Images of geese, clouds, pilots, plane, arrows, cup, airport traffic control tower are property of Mykyta Protsenko and Alex Borysov, if not
stated otherwise (see below). All Rights Reserved.
Other images used:
Slide #5: commons.wikimedia.org/wiki/File:FEMA_-_16381_-_Photograph_by_Bob_McMillan_taken_on_09-28-2005_in_Texas.jpg
- Picture by Bob McMillan, the US federal government work, public domain
Slide #6: www.flickr.com/photos/carbonnyc/3290528875
- Picture by David Goehring. Attribution 2.0 Generic (CC BY 2.0): creativecommons.org/licenses/by/2.0
- changes were made
Slide #7: www.flickr.com/photos/carbonnyc/3290528875
- Picture by Camerafiend. Attribution-ShareAlike 3.0 Unported (CC BY-SA 3.0): creativecommons.org/licenses/by-sa/3.0/deed.en
- no changes were made
Slides ##8, 9: commons.wikimedia.org/wiki/File:Titanic_sinking,_painting_by_Willy_St%C3%B6wer.jpg
- Willy Stöwer. Public domain work of art

136. @aiborisov
@mykyta_p
Images and Licensing
Slides ##8, 10, 13: www.flickr.com/photos/22608787@N00/3200086900
- Picture y Greg Lam Pak Ng. Attribution 2.0 Generic (CC BY 2.0): creativecommons.org/licenses/by/2.0
- no changes were made
Slides ##15-22, 29, 62, 71-74, 83-91, 106-107:
- Blue Game Boy Color by kure: piq.codeus.net/picture/31994/Blue-Game-Boy-Color
- Attribution 3.0 Unported (CC BY 3.0): creativecommons.org/licenses/by/3.0
- changes were made
Slides ##83-91:
- The Sun by Vinicius615: piq.codeus.net/picture/191706/The-Sun
- Attribution 3.0 Unported (CC BY 3.0): creativecommons.org/licenses/by/3.0
- changes were made
Slide #102:
- Picture by Alex Borysov. Attribution 2.0 Generic (CC BY 2.0): creativecommons.org/licenses/by/2.0

137. @aiborisov
@mykyta_p
Images and Licensing
Slides #107, ##131-133: piq.codeus.net/picture/423109/UFO
- UFO by anonymous
- Attribution 3.0 Unported (CC BY 3.0): creativecommons.org/licenses/by/3.0
- no changes were made
Slides ##121, 122: piq.codeus.net/picture/334023/beer
- beer by Investa
- Attribution 3.0 Unported (CC BY 3.0): creativecommons.org/licenses/by/3.0
- changes were made
Slides #121, 122: piq.codeus.net/picture/444498/Beer-Bottle
- Beer Bottle by jacklrj
- Attribution 3.0 Unported (CC BY 3.0): creativecommons.org/licenses/by/3.0
- changes were made

138. @aiborisov
@mykyta_p
Images and Licensing
Slide #126: https://piq.codeus.net/picture/330338/Deal-With-It
- Deal With It by Shiro
- Attribution 3.0 Unported (CC BY 3.0): creativecommons.org/licenses/by/3.0
- changes were made