UC Berkeley - CS10
Giv ing a talk at Ca l’s CS10 - TheBeauty and Joy     of Computing.inst. eecs.berkeley.e   du/~cs10/sp1116 Mar via Twitter...
Follow me! twitter.com/raffi or send a SMS to 40404 follow raffi
To start... What’s a Tweet?
Twitter isthe nature of news.
“Rudimentary communicationamong individuals in real timeallows many to move together asone” - @biz
One WordApril 10, 2008  4:33:23 PM
One Word                 April 10, 2008                   4:33:23 PM And it’s overApril 11, 20081:27:26 PM
Super bowl                 February 2011the Grammys February 2011
TunisiaJanuary 2011               Egypt               January 2011
http://twitter.com/#!/toptweets/status/12483108178
http://twitter.com/#!/Emergency_In_SF/status/29440739442
http://twitter.com/#!/i80chains/status/9726084734
http://twitter.com/#!/remedyoakland/status/29002198672
http://twitter.com/#!/AlbionsOven/status/10015063036
How does Twitter work?
Twitter is just three things:Tweets,relationships between those threeAnd the users, anD Timelines
A Tweet is a simple thing                            140 characters
What’s in a tweet?  {       “id” :   787167620,       “text”   : “Free”,       “date”   : “2008-04-1 1 13:27:26”       “us...
Where do they go?                    Following                    Followed by
Tweets multiply
TWEETs MULTIPLY
HOME TIMELINE                 [27886578700,                  27856404785,                  27727644786,                 27...
Is that it? ... YES! ... sorta ...
things get more complicated at scaleAnd scaling “real-time” is hard
How many Tweets are there?
How many Tweets are there?   140M!                             off the chart
140M tweets       1600 tweets    per day   ≈   per second
140M tweets       1B tweets                  1600 tweets    per day   ≈   per second                      week
Tweets multiply                   8,080 people                  need to see my                           tweet
@ladygagamother mons†er6.8 million followers@justinbieberJustin Bieber5.7 million followers@BarackObama44th President of t...
@ladygagamother mons†er8.8 million followers@raffime!8.0 thousand followers    differs by10     scaling 10 63
106deliveries       6000 deliveries  per minute    ≈   per millisecond
A System is only as strong as itsweakest link
80 MB/sec≈   10 ms/seek
10 ms        100 seeksper seek   ≈   per second
MySQL         Can’t generate IDs fast enoughCentralized and a single point of failure                                     ...
How big are they?                         ≈      1 tweet text       =                         ≈     140 characters        ...
Lots of deliveries
1                   Digraph                           2            Need to represent this                                 ...
Photo used under Creative Commons from jurvetson          Distributed graph databaseflockdb   High rate of CRUD operations...
200M registered users       2006    2008     2010   2011
Using the system   10B API calls       110,000 calls        per day    ≈   per second
REST API         XML/JSON API over HTTPPoll-based system / pseudo real-time                               hosebird        ...
Latency200ms100ms 0ms
Where do we want to be?          Today - 200M people generate ~1600 TPSTomorrow - we want to support half the world and al...
Follow me atQuestions?   twitter.com/raffi
Twitter for CS10 @ Berkeley (Spring 2011)
Twitter for CS10 @ Berkeley (Spring 2011)
Twitter for CS10 @ Berkeley (Spring 2011)
Twitter for CS10 @ Berkeley (Spring 2011)
Twitter for CS10 @ Berkeley (Spring 2011)
Upcoming SlideShare
Loading in …5
×

Twitter for CS10 @ Berkeley (Spring 2011)

1,471
-1

Published on

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,471
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Twitter for CS10 @ Berkeley (Spring 2011)

  1. 1. UC Berkeley - CS10
  2. 2. Giv ing a talk at Ca l’s CS10 - TheBeauty and Joy of Computing.inst. eecs.berkeley.e du/~cs10/sp1116 Mar via Twitter for iPhone from UC Berkeley 277 Cory Hall Berkeley, CA View Tweets at this place
  3. 3. Follow me! twitter.com/raffi or send a SMS to 40404 follow raffi
  4. 4. To start... What’s a Tweet?
  5. 5. Twitter isthe nature of news.
  6. 6. “Rudimentary communicationamong individuals in real timeallows many to move together asone” - @biz
  7. 7. One WordApril 10, 2008 4:33:23 PM
  8. 8. One Word April 10, 2008 4:33:23 PM And it’s overApril 11, 20081:27:26 PM
  9. 9. Super bowl February 2011the Grammys February 2011
  10. 10. TunisiaJanuary 2011 Egypt January 2011
  11. 11. http://twitter.com/#!/toptweets/status/12483108178
  12. 12. http://twitter.com/#!/Emergency_In_SF/status/29440739442
  13. 13. http://twitter.com/#!/i80chains/status/9726084734
  14. 14. http://twitter.com/#!/remedyoakland/status/29002198672
  15. 15. http://twitter.com/#!/AlbionsOven/status/10015063036
  16. 16. How does Twitter work?
  17. 17. Twitter is just three things:Tweets,relationships between those threeAnd the users, anD Timelines
  18. 18. A Tweet is a simple thing 140 characters
  19. 19. What’s in a tweet? { “id” : 787167620, “text” : “Free”, “date” : “2008-04-1 1 13:27:26” “user” : “jamesbuck” }* *not quite, but close!
  20. 20. Where do they go? Following Followed by
  21. 21. Tweets multiply
  22. 22. TWEETs MULTIPLY
  23. 23. HOME TIMELINE [27886578700, 27856404785, 27727644786, 27679412331, 27594550593, 27583976236, 27580246968, 27576503025, 27545926564, 27545450496, 27539864336...
  24. 24. Is that it? ... YES! ... sorta ...
  25. 25. things get more complicated at scaleAnd scaling “real-time” is hard
  26. 26. How many Tweets are there?
  27. 27. How many Tweets are there? 140M! off the chart
  28. 28. 140M tweets 1600 tweets per day ≈ per second
  29. 29. 140M tweets 1B tweets 1600 tweets per day ≈ per second week
  30. 30. Tweets multiply 8,080 people need to see my tweet
  31. 31. @ladygagamother mons†er6.8 million followers@justinbieberJustin Bieber5.7 million followers@BarackObama44th President of the United States5.6 million followers@raffime!5.1 thousand followers
  32. 32. @ladygagamother mons†er8.8 million followers@raffime!8.0 thousand followers differs by10 scaling 10 63
  33. 33. 106deliveries 6000 deliveries per minute ≈ per millisecond
  34. 34. A System is only as strong as itsweakest link
  35. 35. 80 MB/sec≈ 10 ms/seek
  36. 36. 10 ms 100 seeksper seek ≈ per second
  37. 37. MySQL Can’t generate IDs fast enoughCentralized and a single point of failure snowflake Highly available and uncoordinated (10kqps) Compatible with the ecosystem http://github.com/twitter/snowflake
  38. 38. How big are they? ≈ 1 tweet text = ≈ 140 characters 18 MB/min ≈ 200 bytes ≈ 26 GB/day Just tweet text!
  39. 39. Lots of deliveries
  40. 40. 1 Digraph 2 Need to represent this 4 1 2 3 4 31 Matrix2 Naïve implementation is not scalable34
  41. 41. Photo used under Creative Commons from jurvetson Distributed graph databaseflockdb High rate of CRUD operations Complex set arithmetic queries http://github.com/twitter/flockdb
  42. 42. 200M registered users 2006 2008 2010 2011
  43. 43. Using the system 10B API calls 110,000 calls per day ≈ per second
  44. 44. REST API XML/JSON API over HTTPPoll-based system / pseudo real-time hosebird Streaming API Long poll HTTP Near real-time delivery of Tweets
  45. 45. Latency200ms100ms 0ms
  46. 46. Where do we want to be? Today - 200M people generate ~1600 TPSTomorrow - we want to support half the world and all its devices (right now, there are 6B people and 5B phones)
  47. 47. Follow me atQuestions? twitter.com/raffi

×