Going Multiplayer With Kafka With Ben Gamble | Current 2022
Today we’ll walk through building multi-user and multiplayer spaces for games, collaboration, and for creation, leveraging Apache Kafka® for state management, and stream processing to handle conflicts and atomic edits.
We’re not building the metaverse! But as technology matures, ideas jump between disciplines, from ReactJS using ideas from game rendering, to the recent innovations in ECS patterns that borrow heavily from column stores in databases - there’s never been a better time to bring ideas from one sector of software engineering to another. Apache Kafka® makes event management simple so what can we borrow to make it collaborative?
Starting with a simple chat application and working into cursor sharing, collaborative editing and even a multiplayer game, we’ll walk through how to collect and manage user inputs,, and how backing onto an event log allows for version control, undos and time travel. We’ll also explore the various ways you can build up a canonical source of truth in a distributed system, from snapshots, to lockstep sync, to eventual consistency. Along the way we’ll learn a bit about CRDTs (conflict-free replicated data types), mergeable data structures and some of the ways to manage this complexity effectively.
3. ● Tell you all who I am, what I do & who for
● Set the stage for what I am talking about
● Tell you a story
● Explain the issue in more detail
● Talk through some architecture
● Answer some questions
● Chat, Stocks
7. About Me
● A Technology Sommelier
● Worked on everything from MMO’s to medical deivces
● Once started a Google glass AR company
● I code in Rust for more crustacean jokes
86. First we choose some tools
● Fan in → Websockets into Kafka
● Aggregate → Nagle's
● Filter → Remove duplicates
● Process → concat values inside the window
● Fan out → Kafka to more Websockets
90. Change around the algos
● Fan in → Websockets into Kafka
● Aggregate → Nagle's
● Filter → Remove duplicates
● Process → YJS
● Fan out → Kafka to more Websockets
94. Change around the algos
again…
● Fan in → Websockets into Kafka
● Aggregate → Nagle's
● Filter → Spacial decomposition
● Process → Hash and delta compress
● Fan out → Kafka to more Websockets