This document discusses Twitter's use of various big data technologies to handle its massive scale. It describes how Twitter went from its first tweet to sending over 500 million tweets per day. It then discusses the challenges of supporting a real-time social graph at this scale and how Twitter uses FlockDB and the Gizzard framework to shard and query the social graph distributed across multiple servers. It also explains how Twitter uses Hadoop and Storm to handle batch processing and real-time processing of tweets.

1. Term paper presented by:
• Akhtar S.Quereshi
• Anurag Arora
• Divya Gandhi
• Nishant Goyal
DDBMS term paper 1

2. Twitter tale of big data!
3 years, 2 months and 1 day. The time it took from the first Tweet to the billionth Tweet.
1 week. The time it took for users to send a billion Tweets in 2011.
50 million. The average number of Tweets people sent per day, 2010.
140 million. The average number of Tweets people sent per day, February 2011.
177 million. Tweets sent on March 11, 2011.
Half a billion tweets sent per day in Oct 2012.
572,000. Number of new accounts created on March 12, 2011.
460,000. Average number of new accounts per day over February 2011.
DDBMS term paper 2

3. Real-time challenge
DDBMS term paper 3

4. DDBMS term paper 4

5. Agenda of ppt
• Managing social graphs- FlockDB
• Sharding- Gizzard
• Real time data processing/storing:
Hadoop/Storm
DDBMS term paper 5

6. FlockDB- built over MySQL
Maintaining social graph and query processing
DDBMS term paper 6

7. DDBMS term paper 7

8. Challenges
• Timeline needs to rapidly go through the
*following* list of user and quickly display all
their tweets (sorted recency based)
• Answer queries like "What's the intersection of
people I follow and people who are following
President Obama?"
• Handle heavy write traffic, as followers are added or
removed.
DDBMS term paper 8

9. DDBMS term paper 9

10. These features are difficult to
implement in a traditional
relational database.
DDBMS term paper 10

11. What is FlockDB?
• FlockDB is a distributed graph database for storing
adjacency lists.
• Optimized not for graph traversal but very large
adjacency lists and fast read/writes.
• It is able to support:
– a high rate of add/update/remove operations.
– potentially complex set arithmetic queries.
– paging through query result sets containing millions
of entries.
– ability to "archive" and later restore archived edges.
DDBMS term paper 11

12. How FlockDB deals with challenges?
• FlockDB database stores all information as edge
attributes in the graph.
• The four major attributes in the adjacency list
DDBMS term paper 12

13. • Each edge is actually stored twice.
forward: Nick follows Robey at 9:54 today.
backward: Robey is followed by Nick at 9:54 today.
• "Who follows me?" is just as efficient as
"Who do I follow?”
DDBMS term paper 13

14. "What's the intersection of people I follow and
people who are following President Obama?“
.
This can be answered quickly by decomposing it into single-user
query: "Who is following President Obama?“
 Data is partitioned by node, so these queries can
each be answered by a single partition, using an
indexed range query.
 Paging through long result sets is done by using the
position field(timestamp) as a cursor.
DDBMS term paper 14

15. Gizzard Framework is used to query
the flockDB distributed datastore.
And to handle the partitioning layer
DDBMS term paper 15

16. What’s ‘Sharding’
DDBMS term paper 16

17. Sharding
= Partitioning + Replication
The problem is: sharding is difficult.
Determining smart partitioning schemes for
particular kinds of data requires a lot of
thought. And even more difficult is ensuring
that all of the copies of the data are consistent
despite unreliable communication and
occasional computer failures.
DDBMS term paper 17

18. Sharding
The advantages of sharding are:
• High availability
• Faster Queries
How is sharding different than
traditional architectures?
DDBMS term paper 18

19. How is sharding different than
traditional architectures?
• Data are parallelized across many datastores
• Data are more highly available.
• It doesn't use replication
• Data are denormalized
DDBMS term paper 19

20. Gizzard
DDBMS term paper 20

21. Gizzard
Gizzard is a framework that offers a basic
template for solving a certain class of problem.
DDBMS term paper 21

22. Gizzard
Here are some key features of "Gizzard"
 Gizzard supports any datastorage backend
 Gizzard handles partitioning through a forwarding table
 Gizzard is middleware
 Gizzard handles replication through a replication tree
 Gizzard is fault-tolerant
 Gizzard supports migrations
 Gizzard handles write conflicts
DDBMS term paper 22

23. How does
‘Gizzard’ work
DDBMS term paper 23

24. How does it work ?
Gizzard is middleware
It sits “in the middle” between clients (web front-ends like PHP and Ruby
on Rails applications) and the many partitions and replicas of data hence
all the data manipulation flow through Gizzard.
DDBMS term paper 24

25. Architecture
Web/App Server
Gizzard
MySQL
Stateless
DDBMS term paper 25

26. How does it work ?
Gizzard handles partitioning through a
forwarding table
Gizzard handles partitioning by mappings ranges of data to particular
shards.
Stored in a forwarding table
DDBMS term paper 26

27. Partitioning
• Define a function Fun( id )
• Ranges do not have to be
equal
DDBMS term paper 27

28. How does it work ?
Gizzard handles replication through a
replication tree
Each shard referenced in the forwarding table can be either a physical
shard or a logical shard.
A physical shard is a reference to a particular data storage back-end
A logical shard is just a tree of other shards.
DDBMS term paper 28

29. Partitioning
• Logical Shading-Tree
• Define Replication Policy
Read Only, Write Only
Replicate
DDBMS term paper 29

30. How does it work ?
Gizzard is fault-tolerant
Gizzard is designed to avoid any single points of failure.
If a certain replica in a partition has crashed, Gizzard routes requests to
the remaining healthy replicas, bearing in mind the weighting function.
Writes to an unavailable shard are buffered until the shard again becomes
available.
DDBMS term paper 30

31. How does
‘Gizzard’ handle
write conflicts
DDBMS term paper 31

32. Write operations have to be idempotent and
commutative.
Example: A user quickly follows and unfollows me. How
is this write communtative?
Follow and unfollow translate to the same write event
to FlockDB, "set edge state to X". An update applies only
if the state on disk is older than the state in flight. So in
the case of follow then unfollow, it doesn't matter
which one is applied to MySQL first, the unfollow state
will always win as it is more recent.
DDBMS term paper 32

33. How does Gizzard handle write
conflicts ?
Write conflicts are when two manipulations to the same record try to
change the record in differing ways.
Because Gizzard does not guarantee that operations will apply in order.
As described write operations must be both idempotent and commutative
in order to avoid conflicts.
This is actually an easy requirement in many cases than trying to guarantee
ordered delivery of messages with bounded latency and high availability.
DDBMS term paper 33

34. Migration
Migrating from Datastore A to
Datastore A'
DDBMS term paper 34

35. Twitter’s real time data
processing and storage
needs.
What type of data system
does it need?
DDBMS term paper 35

36. DDBMS term paper 36

37. DDBMS term paper 37

38. DDBMS term paper 38

39. DDBMS term paper 39

40. DDBMS term paper 40

41. DDBMS term paper 41

42. DDBMS term paper 42

43. DDBMS term paper 43

44. Hadoop
• Hadoop Distirbuted File System (HDFS)- it breaks
each file you give it into 64- or 128-MB chunks called
blocks and sends them to different machines in the
cluster, replicating each block three times along the
way.
– LZO Compression
• Map reduce workflow system- It breaks analyses
over large sets of data into small chunks which can be
done in parallel across all 100 (say) machines.
Generates the precomputed view on which queries are
executed
DDBMS term paper 44

45. DDBMS term paper 45

46. DDBMS term paper 46

47. DDBMS term paper 47

48. DDBMS term paper 48

49. DDBMS term paper 49

50. DDBMS term paper 50

51. DDBMS term paper 51

52. DDBMS term paper 52

53. DDBMS term paper 53

54. DDBMS term paper 54

55. DDBMS term paper 55

56. DDBMS term paper 56

57. DDBMS term paper 57

58. DDBMS term paper 58

59. DDBMS term paper 59

60. DDBMS term paper 60

61. DDBMS term paper 61

62. Storm topology
DDBMS term paper 62

63. Example Query: streaming word count
DDBMS term paper 63

64. 1.Guaranteed Message processing.
2.Robust Process Management.
3.Fault Detection and Automatic Reassignment.
4.Efficient Message Passing.
USP of STORM:
DDBMS term paper 64

65. Monitoring popular queries
DDBMS term paper 65
Storm topology that
tracks statistics on
search queries
Send to human evaluators
for question AND Amazon’s
Mechanical Turk query
categorizes the query.
Machine learning models
evaluates responses and
then push information to
back end systems

66. 1.Twitter Engineering blog.
2.Github Forums.
References
DDBMS term paper 66

67. Queries please!!
DDBMS term paper 67
Thank you!