How to add reliability and resilience into your code

1. Thank you to our Sponsors
Media Sponsor:

2. System
Reliability and Resilience
and stuff

3. tuple

4. //Initialize customer and
invoice
Initialize(customer, invoice);

5. public void Initialize
(Customer customer, Invoice
invoice)
{
customer.Name = “asdf”;
invoice.Date = DateTime.Now;
}

6. Initialize(customer, invoice);
//did something happen to
customer
// and/or invoice?

7. customer.Name =
InitNameFrom(customer,
invoice);
invoice.Date =
InitDateFrom(customer,
invoice);

8. customer.Name =
GetNameFrom(customer,
invoice);
invoice.Date =
GetDateFrom(customer,
invoice);

9. var results =
Initialize(customer, invoice);
customer.Name = results.Item1;
invoice.Date = results.Item2;

10. public tuple<string, DateTime>
Initialize(customer,
invoice)
{
return new Tuple<string, DateTime>
(“asdf”, DateTime.Now);
}

11. public static bool TryParse
(string s, out DateTime result)
or
public static
tuple<bool, DateTime?> TryParse
(string s)

12. tuple
• Avoid side effects
• Avoid out parameters
• multiple values without a specific type

13. null object

14. private ILogger _logger;
public MyClass(ILogger logger) {
_logger = logger;
}
…
if (_logger != null) {
_logger.Debug(
“it worked on my machine!”);
}

15. null checks for
everyone!

16. forget
one
and…

17. public class NullLogger : ILogger {
public void Debug(string text) {
//do sweet nothing
}
}

18. private ILogger _logger =
new NullLogger();
public MyClass(ILogger logger) {
_logger = logger;
}
…
_logger.Debug(
“it worked on my machine!”);

19. null object
• Can eliminate null checks
• Simple to implement

20. Circuit
Breaker

22. Retry

23. Out of Process
Dependency
N times
Your Application

24. Out of Process
Dependency
N times
*
Y clients

25. =
Denial of
Service Attack

26. Limit the #
of retries

27. N * Y
becomes
5 * Y

28. Y is
still a
problem

30. Circuit
Breaker

32. State Machine
On :: Off

33. On  Off
when not healthy

34. Off  On
manually

35. Get to software
before we ask you
to dance

36. Out of Process
Dependency
Healthy
or
Unhealthy

37. State is
independent of
requestor
Out of Process
Dependency

38. Your Application
Has many
independent
external
dependencies

39. Your Application
Can throttle
itself

40. Your Application
Has a
wait
threshold

41. Your Application
External
Dependency
Circuit Breaker
Threshold = 2
Pause = 10ms
Timeout = 30s
Request State = Closed
Request
Failure (i.e. HTTP 500)
Failure Count = 1
Pause 10ms
Request
Failure (i.e. HTTP 500)
Failure Count = 2
State = Open
OperationFailedException

42. Threshold = 2
Pause = 10ms
Timeout = 30s
Request State = Open
30s has not passed
CircuitBreakerOpenException
Request
30s has not passed
CircuitBreakerOpenException
System can try
to
become
healthy
for 30s
Your Application
External
Dependency
Circuit Breaker

43. Threshold = 2
Pause = 10ms
Timeout = 30s
30s has passed
Your Application
Request State = ½ Open
Request
Failure (i.e. HTTP 500)
Failure Count = 2
State = Open
OperationFailedException
External
Dependency
Circuit Breaker

44. Threshold = 2
Pause = 10ms
Timeout = 30s
30s has passed
Request State = ½ Open
Request
Failure Count =
0
State = Closed
Response
Response
Your Application
External
Dependency
Circuit Breaker

45. Closed
Open
½ Open

46. ½ Open
is like a
manual reset

47. Pause
Timeout

48. Pause
between calls
in the loop

49. Timeout
before you
can call again

50. Exceptions

51. OperationFailed
:
AggregateException

52. CircuitBreakerOpen
:
ApplicationException

53. Don’t Loose
Exception Info

54. Always use
InnerException(s)

55. Threshold = 3
Request State = Closed
Request
Failure (i.e. HTTP 500)
?
Request
Request Failure (i.e. HTTP 500)
Failure Count = 2
Failure Count = 0
State = Closed
Response
Response
Your Application
External
Dependency
Circuit Breaker
Failure Count = 1

56. Segregate
Dependencies

57. circuitBreaker(“database”)
circuitBreaker(“weatherservice”)

58. Dependency type,
endpoint svc,
endpoint

59. Where?

60. Your Application
Circuit Breaker
Proxy
Out of Process
Dependency

61. Watch for
Inception

62. Your Application
Circuit Breaker
Proxy
Web Service
Circuit Breaker
Repository
Database

63. circuit breaker
• retry looping
• slow down attempts
• good neighbour

64. Thank you
Donald Belcham
@dbelcham
donald.belcham@igloocoder.com