The document discusses various types of exceptions in software systems, including systemic, intentional, and transient exceptions, and emphasizes the difference in handling them. It presents resilience strategies such as retries, circuit breakers, and message-based architecture to cope with failures effectively. The document also highlights the importance of idempotency and proper design considerations when implementing retries to ensure system robustness.

1. An exception occurred…
Please try again
LAILA BOUGRIA
@noctovis
@lailabougria

2. Exceptions happen all the time

3.  Systemic exceptions
 Intentional exceptions
 Transient exceptions
All exceptions are not equal

4.  Well-known
 Easy to reproduce
 Identify – Fix – Test
 Techniques: TDD, Integration testing, Acceptance testing
Systemic exceptions

5.  Intentional use of exceptions
 E.g. ValidationException
 Acceptable outcome
 Expect the calling API to handle this
Intentional exceptions

6.  Happen unexpectedly
 Hard to reproduce
 External systems
 Infrastructure
 Some persist longer than others
Transient exceptions

7.  Concurrency exceptions
 Failover of a database cluster
 System overload
 External services
Transient exceptions

8.  Latency
 Hardware recycling
 Throttling
 Network connectivity
Transient exceptions in cloud computing

9.  Hard to identify
 Hard to reproduce
 Hard to test
Transient exceptions

10. An example
Place
order
Success
Failure
Call
support
Move to
competitor

16.  User
 Support engineer
 Software engineer
What more can we do?

17.  Cope with failures
 Without data loss
 Without affecting users
Build resilient systems

18.  Retries
 Retries with exponential back-off
 Circuit breaker
 Fallback
Resilience strategies

19. Retry pattern

20. Retries
 Wrap code in try-catch block
 Exception?
 Repeat!
 Configured amount of times
 Perfect for concurrency issues, flaky connections

21. Immediate retry

22. Retry with exponential back-off

23. Retry with exponential back-off and Jitter

24. Combined immediate and backoff

25. Polly
 Transient-fault-handling library
 Policy-based
 Supports reactive and proactive resilience patterns
 Retry & WaitAndRetry

26. An example with Polly

27. An example with Polly

28. Polly
 Different policies for different needs
 Use the policy where you need retries
 Plug Polly into ASP.NET Core Middleware
 Microsoft.Extensions.Http.Polly package

29. Design considerations
 Nested retries
 Isolate change
 Idempotency

30. Nested retries
 Avoid nested layers of retries
 Exponential number of requests
 Initiating request timeout
 Consider how APIs are implementing retries

31.  Action needs to be a single unit (of work)
 Retry attempts should be independent
 No shared state
 No left-over state
Isolate change

32. Idempotency
 Repeating an action produces equal results
 Duplicate requests will occur
 Multi-infrastructure
 No transactions

33. Remember this?
Place
order
Success
Failure
Call
support
Move to
competitor

34. An example with retries
Place
order
Success
Failure Immediate
retries
Success
Failure Delayed
retries
Success
Failure
Call
support
Move to
competitor

35. All retries failed…

36. We’re having a really bad day

37. At that point…
 Request is lost
 User needs to start over
 They might just give up

38. Adopting a message-based architecture
 Move away from request-response
 Capture request
 Inform the user the request will be handled
 Handle request in dedicated process

39. Handling API
Coupled system
Calling API

40. Handling API
Message-based system
Producer
Queue

41. Successful processing
Consumer
Queue
Ack

42. Queue
Failed processing
Consumer
Nack

43.  Retries don’t affect synchronous path
 Fire-forget nature
 Retry forever
 Business retries
Benefits from a recoverability perspective

44. An example with messaging
Place
order
Send
message
Message in
queue
Process
message
Success
Failure Message
nacked
Message
acked

45. Recoverability with queues
 Only immediate retries
 Increases load on the system
 Retries with exponential backoff
 Persistent failures to error queue

46.  Messaging middleware technology
 High scalability & flexibility
 Support for multiple queuing technologies and data stores
 Supports Outbox and Saga patterns
 Monitoring and debugging tools
NServiceBus

47.  Immediate & delayed retries & custom policies
 Centralized error queue
 Error notifications
 Unrecoverable exceptions
 Automatic rate limiting
 Circuit breaker support
Recoverability in NServiceBus

48. NServiceBus recoverability configuration

49. An example with NServiceBus
Place
order
Send
message
Message in
queue
Process
message
Failure
Success
Immediate
retries
Delayed
retries
Error
queue
Failure
Success
Failure
Success

50. Where do retries apply?

51. Polly outside message handlers

52. Remember idempotency?
 Same message might arrive more than once
 Message deduplication with Outbox
 Messages are guaranteed to be processed exactly once
 Consistency between message and data operations

53. Atomicity problem: zombie record

54. Atomicity problem: ghost message

55. Outbox

56. No Outbox guarantees here

57. Transactional session

58. Don’t ever catch exceptions again!

59. Let’s recap
 Multiple strategies to increase resilience
 Don’t force retries on your users, do it for them!
 Don’t write your own retries
 Embrace asynchronous messaging

60. @noctovis @lailabougria @lailabougria