The document discusses building a distributed datastore over 8 weeks to meet customer demand for storing more data across multiple servers. It outlines potential issues at scale like server crashes and inconsistent data. Possible solutions explored are sharding the data across dedicated servers and using a leader election approach. The project structure involved daily standups and weekly meetings over Zoom. Key challenges addressed were consistency, availability and fault tolerance using techniques like distributed logging, resilience libraries and consensus algorithms. Feedback was overall positive while also noting the challenges of the tight timeframe and working entirely remotely.

1. Building a Distributed
Datastore in 8 Weeks
Yoel Spotts
DevNexus April 2022

2. Background

5. The sales team is in a panic: Our document datastore product has
become so popular that customers are clamoring for bigger and
better. They are storing so much information, they want the ability
to store their data across multiple servers - we need to build a
distributed datastore! We will learn about the challenges of
distributed data, the pros and cons of different approaches and the
tradeoffs we need to make. What should we prioritize? Consistency
of data? Availability? Fault tolerance? Can we have all three? This
will be an exciting journey into the world of distributed systems.

6. Benyamin
Jachter
Seth
Jacobs
Tuvia
Goldstein
Yonatan
Olshin

7. Project

8. Structure
● Standups every day
● Longer meetings once a week
● Quick conversations via slack
● Distributed via zoom
● All work done together
● Pull Requests on GitHub with reviews
and comments

11. Possible Issues at Scale
● Temporary or permanent server crash
● Multiple people trying to access at the same
time leading to inconsistent state
● Update to the same data shared by multiple
consumers
● Bad actors trying to corrupt shared data
● Overload the server with too much traffic; slow
down or crash server
● Run out of storage on the database; downtime
adding more storage or crash server

12. Possible Solutions
● Each book store has their own dedicated
server/virtual private cloud
○ Somewhat solves server crashing issue
○ Solves inconsistent data issue
○ Solves sharing data/ bad actors
○ Cuts down on the traffic; but does not eliminate as
one store may be very popular
○ More disk space available; but does not eliminate
issue

13. Possible Solutions
● Raid system: multiple drives, parts of the data
are on different servers
○ Allows hot swapping; can add and remove without
needing to shut down server
● Optimize database for our usage
○ Helps traffic issues; improves response times
○ Cache data; use an LRU
○ Preconstruct SQL calls
○ Downside: complexity
● Authentication system to restrict updates
● Multiple servers

16. ● Sharding Strategy
● Client Agnostic
● Server Discovery /
Distributed Configuration

19. Client Read - Single Bookstore
Redirect

20. Client Read - Multiple Bookstore
Server Forward

22. Client Write - Single Bookstore
Redirect

23. Client Write - Multiple Bookstores
Inconsistent Data!

24. Read / Write Lock

25. Client Write - Multiple Bookstores
With Leader

26. Leader Election

27. Health Checks

28. Resilience / Fault Tolerance
Resilience4j

29. Resilience / Fault Tolerance
Resilience4j

30. Resilience / Fault Tolerance
Resilience4j

31. Distributed Logging

32. Wrap up
● Hub is single point of failure
● Consensus algorithms (Raft, Paxos, Zab)
● Re-Partitioning data
● Docker
● Authentication/Authorization
● UI

34. What Did I Learn?
● These kids are amazing
● Easy to take knowledge for granted
● Forced me to organize/systemize information
● Fantastic mentorship opportunity
● Be flexible
● Be prepared to drop some important topics
● LOTS of work
● Would I do it again? YES

35. Feedback
● Overall, very positive
● Fantastic way to get introduced to distributed systems
● Breaking down, going step by step was very helpful
● Group meetings were productive
● Amazing to have something concrete to show off at
the end
● Enjoyed being able to present to Elastic co-workers

36. Critical Feedback
● Working all together very challenging
● Find a way to break down work for two teams
● Spring Boot: really nice to bootstrap; seems like
magic after a while
● Struggling with third party libraries
● Include introduction to sockets/ports
● Introduce logging earlier
● More discussion of docker

37. Links
University Summer Projects Page
Project on GitHub
Presentation Video

38. Thank You!