All contents © MuleSoft, LLC
Paul Taylor
Strategic Advisor
JAPAC Customer Success Strategy and Architecture.
UNHAPPY PATH & DEALING WITH BAD
EVENTS
AsyncAPI Online Conference
April 2020
All contents © MuleSoft, LLC
Agenda
1. Events - A Pragmatic View
2. Maintaining Order during Chaos
3. Event Observability
4. Back Pressure - When it just all gets to much
5. Decoupling - Be Careful what you wish for
6. Dead Letter Queue - Now What?
7. Mass Extinction - When Producers go rogue & Saving the Dinosaurs
8. Final Thoughts
2
All contents © MuleSoft, LLC
To me the enterprise canonical data model is like Fight Club – Never mention it
Events
3
Event
State Change
Canonical Data Model Event
State Change Plus
appropriate
enrichment
Event : Address Update
Add Value
Pragmatic view
All contents © MuleSoft, LLC
Easy until it all goes wrong
Event Ordering
4
Event Store
Acc # 42 Event #1
Acc # 10 Event #1
Acc # 42 Event #2
Acc # 42 Event #3
Acc # 16 Event #1
Acc # 10 Event #2
All contents © MuleSoft, LLC
Events - Min
End to End is like ‘Done’ it’s difficult
Event Observability
5
Event Store
-
-
-
-
-
-
End to End - Best
Middleware - OK
Event caused
by another
event
All contents © MuleSoft, LLC
Event
throughput is
dialed back
Not every system can drink from the firehose
Back Pressure
6
Event Store
-
-
-
-
-
-
Returning
503 Status
code
All contents © MuleSoft, LLC
Sometimes it makes you miss tight coupling
Decoupling
7
Event Store
-
-
-
-
-
-
All contents © MuleSoft, LLC
How do you handle it?
Dead Letter Queue (DLQ)
8
Event Store
Acc # 42 Event #1
Acc # 20 Event #1
Acc # 42 Event #2
Acc # 42 Event #3
Acc # 16 Event #1
Acc # 10 Event #2 DLQ
Acc # 42 Event #1
Acc # 42 Event #2
Acc # 42 Event #3
All contents © MuleSoft, LLC
When a Producer becomes an asteroid
Mass Extinction
9
Event Store
-xxxxx
-xxxxx
-xxxxx
-xxxxx
-xxxxx
-xxxxx
All contents © MuleSoft, LLC
Bringing it home
Final Thoughts
10
Event Driven
• Allow Evolution
• Agility is great – Production Stability is even better
Use with a purpose
• Events are just another tool in the tool box
Reduce the friction to work with
• Use common design standards – AsyncAPI !
Things will fail
• Design for failure
Expect the unexpected
• Toilet paper the new gold!
All contents © MuleSoft, LLC
Thank you
Paul Taylor
https://www.linkedin.com/in/paultay/

Unhappy Path & Dealing With Bad Events

  • 1.
    All contents ©MuleSoft, LLC Paul Taylor Strategic Advisor JAPAC Customer Success Strategy and Architecture. UNHAPPY PATH & DEALING WITH BAD EVENTS AsyncAPI Online Conference April 2020
  • 2.
    All contents ©MuleSoft, LLC Agenda 1. Events - A Pragmatic View 2. Maintaining Order during Chaos 3. Event Observability 4. Back Pressure - When it just all gets to much 5. Decoupling - Be Careful what you wish for 6. Dead Letter Queue - Now What? 7. Mass Extinction - When Producers go rogue & Saving the Dinosaurs 8. Final Thoughts 2
  • 3.
    All contents ©MuleSoft, LLC To me the enterprise canonical data model is like Fight Club – Never mention it Events 3 Event State Change Canonical Data Model Event State Change Plus appropriate enrichment Event : Address Update Add Value Pragmatic view
  • 4.
    All contents ©MuleSoft, LLC Easy until it all goes wrong Event Ordering 4 Event Store Acc # 42 Event #1 Acc # 10 Event #1 Acc # 42 Event #2 Acc # 42 Event #3 Acc # 16 Event #1 Acc # 10 Event #2
  • 5.
    All contents ©MuleSoft, LLC Events - Min End to End is like ‘Done’ it’s difficult Event Observability 5 Event Store - - - - - - End to End - Best Middleware - OK Event caused by another event
  • 6.
    All contents ©MuleSoft, LLC Event throughput is dialed back Not every system can drink from the firehose Back Pressure 6 Event Store - - - - - - Returning 503 Status code
  • 7.
    All contents ©MuleSoft, LLC Sometimes it makes you miss tight coupling Decoupling 7 Event Store - - - - - -
  • 8.
    All contents ©MuleSoft, LLC How do you handle it? Dead Letter Queue (DLQ) 8 Event Store Acc # 42 Event #1 Acc # 20 Event #1 Acc # 42 Event #2 Acc # 42 Event #3 Acc # 16 Event #1 Acc # 10 Event #2 DLQ Acc # 42 Event #1 Acc # 42 Event #2 Acc # 42 Event #3
  • 9.
    All contents ©MuleSoft, LLC When a Producer becomes an asteroid Mass Extinction 9 Event Store -xxxxx -xxxxx -xxxxx -xxxxx -xxxxx -xxxxx
  • 10.
    All contents ©MuleSoft, LLC Bringing it home Final Thoughts 10 Event Driven • Allow Evolution • Agility is great – Production Stability is even better Use with a purpose • Events are just another tool in the tool box Reduce the friction to work with • Use common design standards – AsyncAPI ! Things will fail • Design for failure Expect the unexpected • Toilet paper the new gold!
  • 11.
    All contents ©MuleSoft, LLC Thank you Paul Taylor https://www.linkedin.com/in/paultay/