Twitter Gizzard          a framework for sharding                              Python hack-a-thon 2011.02.19              ...
Who am I ?Saturday, February 19, 2011                2
twitter-id: yosukehara                              RIA                       Distributed StorageSaturday, February 19, 20...
What’s ‘Gizzard’ ?Saturday, February 19, 2011                        4
Why db sharding?               Grows with your app (“scales”)               Bottleneck in RDBMS                           ...
Mobile Data Explosion: 75 Exabytes by 2015Saturday, February 19, 2011                                        6
1. Sharding - Write partition         2. Cache         3. On-Memory FSSaturday, February 19, 2011              7
Sharding                                = Partitioning + Replication             “                The problem is: sharding...
Partitioning                              Spread keyspace across many nodes                              Scale reads and w...
Partitioning                              Spread keyspace across many nodes                              Scale reads and w...
Replication                              Keep multiple copies of same data                              Scale reads, survi...
Replication                              Keep multiple copies of same data                              Scale reads, survi...
Used by                 FlockDB, Haplocheirus                  social graph-db   message vector cacheSaturday, February 19...
Restriction                              NOT able to ‘Table JOIN’                              NOT able to ‘Record Groupin...
Architecture          Web/App Server                    Gizzard                              MySQLSaturday, February 19, 2...
Architecture          Web/App Server                              Gizzard   Gizzard made by ‘Scala’                       ...
Architecture          Web/App Server                              Gizzard   Stateless                              MySQLSa...
Partitioning                              Define a function Fun( id )                              Map ranges of co-domain ...
Partitioning                              Logical Shading-Tree                                                            ...
Architecture          Web/App Server                              Gizzard                              MySQL MySQL, Lucene...
Architecture          Web/App Server                                        Apache Thrift                              Giz...
Thrift          namespace java com.twitter.rowz.thrift          namespace rb Rowz          # columns          struct Row {...
sharding codesSaturday, February 19, 2011                    23
Plug               the Gizzard Demo app, Rowz               <http://bit.ly/bJXuC7>               MySQL Spider (created by ...
SpiderSaturday, February 19, 2011   25
Thank you for listeningSaturday, February 19, 2011                  26
Upcoming SlideShare
Loading in …5
×

Introduction of Twitter Gizzard

11,489 views

Published on

about Twitter Gizzard

Published in: Technology
0 Comments
25 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
11,489
On SlideShare
0
From Embeds
0
Number of Embeds
442
Actions
Shares
0
Downloads
149
Comments
0
Likes
25
Embeds 0
No embeds

No notes for slide

Introduction of Twitter Gizzard

  1. Twitter Gizzard a framework for sharding Python hack-a-thon 2011.02.19 Yosuke Hara {twiter_id, yosukehara} Software DeveloperSaturday, February 19, 2011 1
  2. Who am I ?Saturday, February 19, 2011 2
  3. twitter-id: yosukehara RIA Distributed StorageSaturday, February 19, 2011 3
  4. What’s ‘Gizzard’ ?Saturday, February 19, 2011 4
  5. Why db sharding? Grows with your app (“scales”) Bottleneck in RDBMS Almost always DISK I/OSaturday, February 19, 2011 5
  6. Mobile Data Explosion: 75 Exabytes by 2015Saturday, February 19, 2011 6
  7. 1. Sharding - Write partition 2. Cache 3. On-Memory FSSaturday, February 19, 2011 7
  8. Sharding = Partitioning + Replication “ The problem is: sharding is difficult. Determining smart partitioning schemes for particular kinds of data requires a lot of thought. And even more difficult is ensuring that all of the copies of the data are consistent despite unreliable communication and occasional computer failures.Saturday, February 19, 2011 8
  9. Partitioning Spread keyspace across many nodes Scale reads and writesSaturday, February 19, 2011 9
  10. Partitioning Spread keyspace across many nodes Scale reads and writes scale-outSaturday, February 19, 2011 10
  11. Replication Keep multiple copies of same data Scale reads, survive failuresSaturday, February 19, 2011 11
  12. Replication Keep multiple copies of same data Scale reads, survive failures high availabilitySaturday, February 19, 2011 12
  13. Used by FlockDB, Haplocheirus social graph-db message vector cacheSaturday, February 19, 2011 13
  14. Restriction NOT able to ‘Table JOIN’ NOT able to ‘Record Grouping’ Client : App retrieve data >> Joins / GroupingSaturday, February 19, 2011 14
  15. Architecture Web/App Server Gizzard MySQLSaturday, February 19, 2011 15
  16. Architecture Web/App Server Gizzard Gizzard made by ‘Scala’ MySQLSaturday, February 19, 2011 16
  17. Architecture Web/App Server Gizzard Stateless MySQLSaturday, February 19, 2011 17
  18. Partitioning Define a function Fun( id ) Map ranges of co-domain of Fun to shards Ranges do not have to be equal [ id ] [ shard ]Saturday, February 19, 2011 18
  19. Partitioning Logical Shading-Tree replicate Define Replication Policy Read Only, Write Only Replicate replicate data-store data-store data-store data-storeSaturday, February 19, 2011 19
  20. Architecture Web/App Server Gizzard MySQL MySQL, Lucene, Redis, ...Saturday, February 19, 2011 20
  21. Architecture Web/App Server Apache Thrift Gizzard MySQLSaturday, February 19, 2011 21
  22. Thrift namespace java com.twitter.rowz.thrift namespace rb Rowz # columns struct Row { 1: i64 id 2: string name 3: i32 created_at 4: i32 updated_at 5: i32 state } # exception exception RowzException { 1: string description } # RPC service service Rowz { i64 create(1: string name, 2: i32 at) throws(1: RowzException ex) void destroy(1: Row row, 2: i32 at) throws(1: RowzException ex) Row read(1: i64 id) throws(1: RowzException ex) }Saturday, February 19, 2011 22
  23. sharding codesSaturday, February 19, 2011 23
  24. Plug the Gizzard Demo app, Rowz <http://bit.ly/bJXuC7> MySQL Spider (created by shiba-san) <http://bit.ly/al2q1j> <http://bit.ly/a1HK2n>Saturday, February 19, 2011 24
  25. SpiderSaturday, February 19, 2011 25
  26. Thank you for listeningSaturday, February 19, 2011 26

×