• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Introduction of Twitter Gizzard
 

Introduction of Twitter Gizzard

on

  • 8,387 views

about Twitter Gizzard

about Twitter Gizzard

Statistics

Views

Total Views
8,387
Views on SlideShare
7,998
Embed Views
389

Actions

Likes
19
Downloads
124
Comments
0

14 Embeds 389

http://d.hatena.ne.jp 204
http://blog.newitfarmer.com 82
http://wiki.simplexi.com 46
http://shiumachi.hatenablog.com 23
http://www.scoop.it 12
http://paper.li 5
http://cncc.bingj.com 4
https://twitter.com 3
http://webcache.googleusercontent.com 3
http://blog.cooooo.co 2
http://www.iweb34.com 2
https://si0.twimg.com 1
http://wiki.simplexi.com:8080 1
http://twitter.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Introduction of Twitter Gizzard Introduction of Twitter Gizzard Presentation Transcript

    • Twitter Gizzard a framework for sharding Python hack-a-thon 2011.02.19 Yosuke Hara {twiter_id, yosukehara} Software DeveloperSaturday, February 19, 2011 1
    • Who am I ?Saturday, February 19, 2011 2
    • twitter-id: yosukehara RIA Distributed StorageSaturday, February 19, 2011 3
    • What’s ‘Gizzard’ ?Saturday, February 19, 2011 4
    • Why db sharding? Grows with your app (“scales”) Bottleneck in RDBMS Almost always DISK I/OSaturday, February 19, 2011 5
    • Mobile Data Explosion: 75 Exabytes by 2015Saturday, February 19, 2011 6
    • 1. Sharding - Write partition 2. Cache 3. On-Memory FSSaturday, February 19, 2011 7
    • Sharding = Partitioning + Replication “ The problem is: sharding is difficult. Determining smart partitioning schemes for particular kinds of data requires a lot of thought. And even more difficult is ensuring that all of the copies of the data are consistent despite unreliable communication and occasional computer failures.Saturday, February 19, 2011 8
    • Partitioning Spread keyspace across many nodes Scale reads and writesSaturday, February 19, 2011 9
    • Partitioning Spread keyspace across many nodes Scale reads and writes scale-outSaturday, February 19, 2011 10
    • Replication Keep multiple copies of same data Scale reads, survive failuresSaturday, February 19, 2011 11
    • Replication Keep multiple copies of same data Scale reads, survive failures high availabilitySaturday, February 19, 2011 12
    • Used by FlockDB, Haplocheirus social graph-db message vector cacheSaturday, February 19, 2011 13
    • Restriction NOT able to ‘Table JOIN’ NOT able to ‘Record Grouping’ Client : App retrieve data >> Joins / GroupingSaturday, February 19, 2011 14
    • Architecture Web/App Server Gizzard MySQLSaturday, February 19, 2011 15
    • Architecture Web/App Server Gizzard Gizzard made by ‘Scala’ MySQLSaturday, February 19, 2011 16
    • Architecture Web/App Server Gizzard Stateless MySQLSaturday, February 19, 2011 17
    • Partitioning Define a function Fun( id ) Map ranges of co-domain of Fun to shards Ranges do not have to be equal [ id ] [ shard ]Saturday, February 19, 2011 18
    • Partitioning Logical Shading-Tree replicate Define Replication Policy Read Only, Write Only Replicate replicate data-store data-store data-store data-storeSaturday, February 19, 2011 19
    • Architecture Web/App Server Gizzard MySQL MySQL, Lucene, Redis, ...Saturday, February 19, 2011 20
    • Architecture Web/App Server Apache Thrift Gizzard MySQLSaturday, February 19, 2011 21
    • Thrift namespace java com.twitter.rowz.thrift namespace rb Rowz # columns struct Row { 1: i64 id 2: string name 3: i32 created_at 4: i32 updated_at 5: i32 state } # exception exception RowzException { 1: string description } # RPC service service Rowz { i64 create(1: string name, 2: i32 at) throws(1: RowzException ex) void destroy(1: Row row, 2: i32 at) throws(1: RowzException ex) Row read(1: i64 id) throws(1: RowzException ex) }Saturday, February 19, 2011 22
    • sharding codesSaturday, February 19, 2011 23
    • Plug the Gizzard Demo app, Rowz <http://bit.ly/bJXuC7> MySQL Spider (created by shiba-san) <http://bit.ly/al2q1j> <http://bit.ly/a1HK2n>Saturday, February 19, 2011 24
    • SpiderSaturday, February 19, 2011 25
    • Thank you for listeningSaturday, February 19, 2011 26