Adopting Actors: An epic tail of loss and learning
•Download as PPTX, PDF•
1 like•250 views
Adopting actors was a journey and some of our early assumptions would create issues later on. Slowly we realised we had not fully embraced the let-it-crash philosophy, did not understand how the atomic nature of actors effects data sovereignty or how parts of our application would have to be eventually consistent. In this talk I will share some of our mistakes, how we arrived at those mistakes and how we eventually resolved them. I explore the reasoning behind our mistakes and the lessons we discovered, so hopefully you can recognise and avoid them. https://www.youtube.com/watch?v=4R1u7EEDn8Y
Recommended
Recommended
More Related Content
Similar to Adopting Actors: An epic tail of loss and learning
Similar to Adopting Actors: An epic tail of loss and learning (10)
Recently uploaded
Recently uploaded (20)
Adopting Actors: An epic tail of loss and learning
- 1. Adopting actors An epic tail of loss and learning
- 3. Workday
- 4. Growth 2013 2014 2015 2016
- 5. Cloud Master Launch tasks Assign to agents
- 6. Cloud Master Launch tasks Assign to agents
- 8. Service Growth in millions of tasks per month 0 5 10 15 20 Print Large Small Batch
- 9. Why Akka?
- 12. Parent Config Child Snapshots Changes Message flow: Ensure messages follow a consistent path
- 13. Parent Config Child Snapshots Changes Creation: Assume actor is recovering from failure (state machine)
- 14. Anti-patterns
- 15. God Class
- 16. Movie Star
- 17. Pool Agent State Agent Agent Agent Agent Queu e
- 18. Movie Star Too much state • Hard to reason about • Too many messages in flight • Hard to recover • Bad concurrency
- 19. Split Brain
- 20. Pool Agent State Agent Agent Agent Agent
- 21. Duplicate state Single source of truth • Synchronizing state is hard • Failure causes –State out of sync –Causes more failure
- 22. Split Brain
- 23. Pool Agent State Agent Agent Agent Agent Task
- 24. Passing responsibility Seems simple at first • Do not always know who is in control • Both actors updating the same row • Creates race conditions
- 25. Can you let it crash?
- 26. Pool Agent State Agent Agent Agent Agent Can you let it crash?
- 27. Lessons
- 28. Test for resilience • Chaos Marmoset • Unit test recovery • Destructive system test
- 30. Sovereignty One actor • One row • One shard • One table Otherwise failure hard to handle
- 31. Atomicity Actors Atomic receive method State not shared Comms async messages Not nestable Mutex Atomic scope State is shared Comms via mutable state Nestable (ACID)
- 33. Pool Agent State Agent Agent Agent Agent Atomicity
- 35. Lessons - Atomicity and Consistency - Actor modeling ≠ Object modeling - Test for Resilience not robustness - Refactor Early
Editor's Notes
- Principle Engineer – Workday’s Grid Cloud Master team. – Who is workday
- Finance and Human Capital Management – ERP Vendor – 100% in the cloud – all customers on a single version
- Fiscal 2016 Total Revenue of $1.16 billion, up 48% year over year Over 5000 employees, over 500 employees in Dublin 2016: Best Workplaces in Ireland, Great Place to Work Institute (#2 for large companies) 2016: 10 Best Large Workplaces in Tech, Fortune (#2)
- provide elastic grid – other services Reliable execution of background tasks or Jobs – pdf printing to payrole Cloudmaster - Agents - Schedule and assign to Agents
- 5 pools of agents Different types of task, memory size, execution speed
- 5 data centers Secure Reliable Safe Isolated – fairness Scalable - Efficient
- This talk is about the lessons I learned migrating a multithreaded java server application to Akka. To support this growth we need to move to stateful services -- Why
- Actor model of concurrency: Safer (no deadlocks) Easier to reason about Easier to test Better distribution Easier scalability Then Scala because of akka – key selling point
- Trying to avoid two way relationship (coupling – mutability) Static State should be immutable
- Trying to avoid two way relationship (coupling – mutability) Static State should be immutable
- Trying to avoid two way relationship (coupling – mutability) Static State should be immutable
- Everyone knows about the God class – threading and mutexes make this worse
- Some are big - Marlon Brando – some are small Robert Downey Junior - me Even when small - entourage
- AgentPoolActor - Responsible for – Agent actors – Queue of tasks – and their assignments Decomposed into separate classes and traits - Still one actor with an entourage
- Also drives more bad decisions
- AgentPoolActor and AgentStateActor External DB changes – sending notifications – message loss – recovery Caused by movie star – Thought problem was stream of events were inconsistent – fix that State Inconsistent – failure – production outage
- … Beauty of split brains
- AgentPoolActor takes job from the Queue Assigns it to an Agent Agent might fail and put it back Pool or Agent might own the job - Cannot reliably find the job EG Cancel Job
- Who - When
- PoolActor has decided to assign task to an agent Async message to StateActor – PoolActor must ensure agent not reused – before reply What if reply timesout??? Crash - Can I guarantee consistency – what happens to the job?
- Chaos Marmoset base actor overrides the unhandled method Messages can cause failures or delays
- Horizontal scalability by pushing all state into the database Actors are about data – Actors are Stateful – Impedance Stateless services cannot update the same data as actor
- Autonomy – single responsibility If your actors write to the database
- We want agent assignments to be consistent
- Banking Transactions ACID? No - Suspense Account – Reconciliation – Compensating transactions Must handle failure cases