Opportunities to Improve System Reliability and Resilience Donald Belcham .NET Conf UY 2014 http://netconf.uy

1. System
Reliability and Resilience
and stuff

2. Some things need to be
cleared up first

3. http://en.wikipedia.org/wiki/Vedette_(cabaret)

4. tuple

5. //Initialize customer and
invoice
Initialize(customer, invoice);

6. public void Initialize
(Customer customer, Invoice
invoice)
{
customer.Name = “asdf”;
invoice.Date = DateTime.Now;
}

7. Initialize(customer, invoice);
//did something happen to
customer
// and/or invoice?

8. customer.Name =
InitNameFrom(customer,
invoice);
invoice.Date =
InitDateFrom(customer,
invoice);

9. customer.Name =
GetNameFrom(customer,
invoice);
invoice.Date =
GetDateFrom(customer,
invoice);

10. var results =
Initialize(customer, invoice);
customer.Name = results.Item1;
invoice.Date = results.Item2;

11. public tuple<string, DateTime>
Initialize(customer,
invoice)
{
return new Tuple<string, DateTime>
(“asdf”, DateTime.Now);
}

12. public static bool TryParse
(string s, out DateTime result)
or
public static
tuple<bool, DateTime?> TryParse
(string s)

13. tuple
• Avoid side effects
• Avoid out parameters
• multiple values without a specific type

14. null object

15. private ILogger _logger;
public MyClass(ILogger logger) {
_logger = logger;
}
…
if (_logger != null) {
_logger.Debug(
“it worked on my machine!”);
}

16. null checks for
everyone!

17. forget
one
and…

18. public class NullLogger : ILogger {
public void Debug(string text) {
//do sweet nothing
}
}

19. private ILogger _logger =
new NullLogger();
public MyClass(ILogger logger) {
_logger = logger;
}
…
_logger.Debug(
“it worked on my machine!”);

20. null object
• Can eliminate null checks
• Simple to implement

21. Circuit
Breaker

23. Retry

24. Out of Process
Dependency
N times
Your Application

25. Out of Process
Dependency
N times
*
Y clients

26. =
Denial of
Service Attack

27. Limit the #
of retries

28. N * Y
becomes
5 * Y

29. Y is
still a
problem

31. Circuit
Breaker

33. State Machine
On :: Off

34. On  Off
when not healthy

35. Off  On
manually

36. Get to software
before we ask you
to dance

37. Out of Process
Dependency
Healthy
or
Unhealthy

38. State is
independent of
requestor
Out of Process
Dependency

39. Your Application
Has many
independent
external
dependencies

40. Your Application
Can throttle
itself

41. Your Application
Has a
wait
threshold

42. Your Application
External
Dependency
Circuit Breaker
Threshold = 2
Pause = 10ms
Timeout = 30s
Request State = Closed
Request
Failure (i.e. HTTP 500)
Failure Count = 1
Pause 10ms
Request
Failure (i.e. HTTP 500)
Failure Count = 2
State = Open
OperationFailedException

43. Threshold = 2
Pause = 10ms
Timeout = 30s
Request State = Open
30s has not passed
CircuitBreakerOpenException
Request
30s has not passed
CircuitBreakerOpenException
System can try
to
become
healthy
for 30s
Your Application
External
Dependency
Circuit Breaker

44. Threshold = 2
Pause = 10ms
Timeout = 30s
30s has passed
Your Application
Request State = ½ Open
Request
Failure (i.e. HTTP 500)
Failure Count = 2
State = Open
OperationFailedException
External
Dependency
Circuit Breaker

45. Threshold = 2
Pause = 10ms
Timeout = 30s
30s has passed
Request State = ½ Open
Request
Failure Count =
0
State = Closed
Response
Response
Your Application
External
Dependency
Circuit Breaker

46. Closed
Open
½ Open

47. ½ Open
is like a
manual reset

48. Pause
Timeout

49. Pause
between calls
in the loop

50. Timeout
before you
can call again

51. Exceptions

52. OperationFailed
:
AggregateException

53. CircuitBreakerOpen
:
ApplicationException

54. Don’t Loose
Exception Info

55. Always use
InnerException(s)

56. Threshold = 3
Request State = Closed
Request
Failure (i.e. HTTP 500)
?
Request
Request Failure (i.e. HTTP 500)
Failure Count = 2
Failure Count = 0
State = Closed
Response
Response
Your Application
External
Dependency
Circuit Breaker
Failure Count = 1

57. Segregate
Dependencies

58. circuitBreaker(“database”)
circuitBreaker(“weatherservice”)

59. Dependency type,
endpoint svc,
endpoint

60. Where?

61. Your Application
Circuit Breaker
Proxy
Out of Process
Dependency

62. Watch for
Inception

63. Your Application
Circuit Breaker
Proxy
Web Service
Circuit Breaker
Repository
Database

64. circuit breaker
• retry looping
• slow down attempts
• good neighbour

65. ¡Muchas gracias!

66. gracias
Donald Belcham
@dbelcham
donald.belcham@igloocoder.com