Going Multiplayer With Kafka With Ben Gamble | Current 2022 Today we’ll walk through building multi-user and multiplayer spaces for games, collaboration, and for creation, leveraging Apache Kafka® for state management, and stream processing to handle conflicts and atomic edits. We’re not building the metaverse! But as technology matures, ideas jump between disciplines, from ReactJS using ideas from game rendering, to the recent innovations in ECS patterns that borrow heavily from column stores in databases - there’s never been a better time to bring ideas from one sector of software engineering to another. Apache Kafka® makes event management simple so what can we borrow to make it collaborative? Starting with a simple chat application and working into cursor sharing, collaborative editing and even a multiplayer game, we’ll walk through how to collect and manage user inputs,, and how backing onto an event log allows for version control, undos and time travel. We’ll also explore the various ways you can build up a canonical source of truth in a distributed system, from snapshots, to lockstep sync, to eventual consistency. Along the way we’ll learn a bit about CRDTs (conflict-free replicated data types), mergeable data structures and some of the ways to manage this complexity effectively.

1. Going Multiplayer with
Kafka

2. WARNING!!
Puns mostly intended

3. ● Tell you all who I am, what I do & who for
● Set the stage for what I am talking about
● Tell you a story
● Explain the issue in more detail
● Talk through some architecture
● Answer some questions
● Chat, Stocks

4. Carcinization

5. Carcinization

6. Carcinization

7. About Me
● A Technology Sommelier
● Worked on everything from MMO’s to medical deivces
● Once started a Google glass AR company
● I code in Rust for more crustacean jokes

8. What is multiplayer?

9. What is multiplayer?

10. Whats multiplayer

11. What is multiplayer?

12. What is multiplayer?
A system, tool or setup that lets
2 or more people interact with each other
In a way that improves the experience

13. Quick history - 1968

14. Quick history - 1970s

15. Quick history -1991

16. Quick history 1993 -1997

17. What things need to be
multiplayer?

18. What things need to be
multiplayer?

19. What things need to be
multiplayer?

20. What makes things
multiplayer?

21. Shared State &
Collaborative Experience

22. Which is
Remote and Concurrent

23. Shared State
● Share a starting point
● Distribute changes
● Achieve consensus
● Deal with divergence
● Be able to continue where you left of

24. State
● Share a starting point

25. State
● Share a starting point
● Distribute changes

26. State
● Share a starting point
● Distribute changes
● Achieve consensus

27. State
● Share a starting point
● Distribute changes
● Achieve consensus
● Deal with divergence

28. State
● Share a starting point
● Distribute changes
● Achieve consensus
● Deal with divergence
● Be able to continue where you left of

29. What makes a good
multiplayer
Experience?

30. Things are
● In Sync

31. Things are
● In Sync
● In Order

32. Things are
● In Sync
● In Order
● Predictable

33. The problems arise
when they are not….

34. So how to people
manage all this?

35. 128 players at high
speed in 2001

37. Multiplayer strategies in
games and their
collaborative counterparts

38. Multiplayer strategies in
games and their
collaborative counterparts

39. Turn based

40. Turn based

41. Similar to pairing

42. Lock-step

43. Similar to text editing

44. State sync with roll back

46. every 16ms

47. CRDTs are close
though
Closest equivalent is
an operational
Transform

48. CRDTs are magic though?

49. No way to govern data

50. Eventually consistent is
unbounded

51. What makes else this
hard?

52. This pattern no longer
holds up

53. Crossed streams == N Squared

54. How do we fix this?

55. A way to connect Large
numbers of things

56. A place to work on that
data

57. Something to plug it all together

58. Let’s talk about
Pub/Sub

59. Publisher ➔ Broker ➔ Subscriber

60. Publisher ➔ Broker ➔ Subscriber

61. Publisher ➔ Broker ➔ Subscriber

62. Publisher ➔ Broker ➔ Subscriber

64. Stream processing

65. Let’s talk about
Websockets

67. TL;DR

68. Its a Fast,
Highly Available,
Websocket PubSub
built to Scale
It gets Kafka beyond the firewall

69. Lets look at Chat

70. Some of it is not good…

71. With Stream processing

72. We get a central authority

73. Edits are just events with
the same Key

74. Pubsub also give other wins

75. Pubsub also give other wins

76. Pubsub also give other wins

77. ~8 Lines of code later

78. Shall we try it?

79. https://kafka-chat.vercel.app/

80. The other thing
about chat…

81. There is LOTS OF IT
35,995

82. Kafka scales

83. Kafka scales reliably

84. Now we have the core
model
● Fan in
● Aggregate
● Filter
● Process
● Fan out

85. Let’s make chat into
cursor sharing

86. First we choose some tools
● Fan in → Websockets into Kafka
● Aggregate → Nagle's
● Filter → Remove duplicates
● Process → concat values inside the window
● Fan out → Kafka to more Websockets

87. ~10 lines of code later….

88. Scaling cursor sharing

89. But wait…
There’s more

90. Change around the algos
● Fan in → Websockets into Kafka
● Aggregate → Nagle's
● Filter → Remove duplicates
● Process → YJS
● Fan out → Kafka to more Websockets

91. A bit like this

92. You get CRDTS with
central authority &
History

93. But wait…
There’s more

94. Change around the algos
again…
● Fan in → Websockets into Kafka
● Aggregate → Nagle's
● Filter → Spacial decomposition
● Process → Hash and delta compress
● Fan out → Kafka to more Websockets

95. You can do all kinds of things

96. Shall we try it?

97. https://crab-craft-kafka.vercel.app/

99. Thank you
Questions?
@BenGamble7