0
Introduction to Riak                           Python hack-a-thon 2011.10.15                                Yosuke Hara {t...
Who am I ?Saturday, October 15, 11                2
twitter-id: yosukehara                               RIA                           Distributed StorageSaturday, October 15...
What’s ‘Riak’ ?                            bashoSaturday, October 15, 11                     4
KVS + ExtraSaturday, October 15, 11                 5
Basho Bitcask                                or                             LevelDB                             (Google, f...
Architecture             Erlang/OTP Runtime                           Client APIs          HTTP                Protocol Bu...
The horizontal scaling bitsSaturday, October 15, 11                 8
Inspired Amazon’s Dynamo (2007)                KVS (Key-Value Storage) + Extra                Masterless, P2P-Replication ...
Inspired Amazon’s Dynamo              http://www.allthingsdistributed.com/2007/10/amazons_dynamo.html                     ...
KVS (Key-Value-Store)                           Simple operations - get, put, delete                           Value is mo...
P2P - MasterlessSaturday, October 15, 11   12
P2P - Masterless                           NOT SPOFSaturday, October 15, 11              12
Consistent Hashing                           Virtual Node ID = SHA1( $KEY )                                            Vir...
Consistent Hashing                           Virtual Node ID = SHA1( $KEY )                                 Virtual Node  ...
For Parallelism                              # of VNodes == Maximum Concurrent Reqs                           For Rebalanc...
Quorum       Every request contacts all replicas of key       N : number of replicas (default 3)       R : read quorum    ...
Distributed Data                                          (1) PUT File   (3) if # of replicas needed                      ...
Hinted Handoff                           X                                               (1) PUT File                     ...
Eventual ConsistencySaturday, October 15, 11                          18
CAP Theorem                           Consistency   Availability                              Partition ToleranceSaturday,...
The doc-database bitsSaturday, October 15, 11               20
Store objects as JSON or any format                Link between objects                No SQL - Query with Javascript Map-...
The operations-friendly bitsSaturday, October 15, 11                    22
Web-shaped storage                      Store data in its original format                      Using HTTP                 ...
DEMO                           bashoSaturday, October 15, 11             24
PUT    curl -X PUT       -H "x-riak-index-service_bin: news"       -H "x-riak-index-range_int: 201110151300"       -H "Con...
GET    curl http://127.0.0.1:8098/demo/news/log_001 | python -mjson.tool    {          "key": "news/photo/20111015_small.j...
Seconday Index (2I)    curl http://127.0.0.1:8098/buckets/demo/index/service_bin/news        | python -mjson.tool    {    ...
Map Reduce  curl -H "Content-Type: application/json" http://127.0.0.1:8098/mapred       --data @-<<EOF  {"inputs":{"bucket...
basho_benchSaturday, October 15, 11                 29
basho_bench                           Depend on Erlang                           Able to be ‘Easy Operation’              ...
Benchmartk ResultSaturday, October 15, 11                       31
basho_bench config         {mode, max}.         {duration, 15}.         {concurrent, 5}.         {driver, basho_bench_drive...
Thank you for listeningSaturday, October 15, 11              33
Upcoming SlideShare
Loading in...5
×

Introduction to Riak

3,532

Published on

Published in: Technology, News & Politics
0 Comments
8 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,532
On Slideshare
0
From Embeds
0
Number of Embeds
8
Actions
Shares
0
Downloads
83
Comments
0
Likes
8
Embeds 0
No embeds

No notes for slide

Transcript of "Introduction to Riak"

  1. 1. Introduction to Riak Python hack-a-thon 2011.10.15 Yosuke Hara {twiter_id, yosukehara} Software DeveloperSaturday, October 15, 11 1
  2. 2. Who am I ?Saturday, October 15, 11 2
  3. 3. twitter-id: yosukehara RIA Distributed StorageSaturday, October 15, 11 3
  4. 4. What’s ‘Riak’ ? bashoSaturday, October 15, 11 4
  5. 5. KVS + ExtraSaturday, October 15, 11 5
  6. 6. Basho Bitcask or LevelDB (Google, from 1.0)Saturday, October 15, 11 6
  7. 7. Architecture Erlang/OTP Runtime Client APIs HTTP Protocol Buffers Erlang local client Request Coordination get put delete map-reduce consistent hashing handoff gossip membership node-liveness buckets Riak Core vnode master vnodes Workers storage backend Riak KVSaturday, October 15, 11 7
  8. 8. The horizontal scaling bitsSaturday, October 15, 11 8
  9. 9. Inspired Amazon’s Dynamo (2007) KVS (Key-Value Storage) + Extra Masterless, P2P-Replication Consistent Hashing Failover - Quorums, Hinted handoff Eventual ConsistencySaturday, October 15, 11 9
  10. 10. Inspired Amazon’s Dynamo http://www.allthingsdistributed.com/2007/10/amazons_dynamo.html “The reliability and scalability of a system is dependent on how its application state is managed.” - Werner VogelsSaturday, October 15, 11 10
  11. 11. KVS (Key-Value-Store) Simple operations - get, put, delete Value is mostly opaque (some metadata) Extras MapReduce (depends on Javascript) Links Full-text search (Optional - Riak Search) Secondary Indexes (from ver 1.0)Saturday, October 15, 11 11
  12. 12. P2P - MasterlessSaturday, October 15, 11 12
  13. 13. P2P - Masterless NOT SPOFSaturday, October 15, 11 12
  14. 14. Consistent Hashing Virtual Node ID = SHA1( $KEY ) Virtual Node “Hash Ring” 2^160 / 32 Node-0 Node-1 Node-2 Node-3Saturday, October 15, 11 13
  15. 15. Consistent Hashing Virtual Node ID = SHA1( $KEY ) Virtual Node Logical subdivision of the cluster “Hash Ring” 2^160 / 32 Node-0 Node-1 Node-2 Node-3Saturday, October 15, 11 13
  16. 16. For Parallelism # of VNodes == Maximum Concurrent Reqs For Rebalancing the Cluster Smallest block that can be shifted to a new node For Resiliance The system restarts failed VNodesSaturday, October 15, 11 14
  17. 17. Quorum Every request contacts all replicas of key N : number of replicas (default 3) R : read quorum W : write quorum Able to adjust to “Consistency Level”Saturday, October 15, 11 15
  18. 18. Distributed Data (1) PUT File (3) if # of replicas needed for a successful Write > 2 # of replicas = 3 (4) return “ok” # of successful WRITEs = 2 Node-0 Node-1 Node-2 Node-3 (2) Replicate - Router on each nodeSaturday, October 15, 11 16
  19. 19. Hinted Handoff X (1) PUT File X (3) if # of replicas needed X for a successful Write > 2 # of replicas = 3 X (4) return “ok” X # of successful WRITEs = 2 X Node-0 Node-1 X X Node-2 (2) Replicate Node-3 X (5) RecoverSaturday, October 15, 11 17
  20. 20. Eventual ConsistencySaturday, October 15, 11 18
  21. 21. CAP Theorem Consistency Availability Partition ToleranceSaturday, October 15, 11 19
  22. 22. The doc-database bitsSaturday, October 15, 11 20
  23. 23. Store objects as JSON or any format Link between objects No SQL - Query with Javascript Map-ReduceSaturday, October 15, 11 21
  24. 24. The operations-friendly bitsSaturday, October 15, 11 22
  25. 25. Web-shaped storage Store data in its original format Using HTTP Load balancing Caches No special-nodes - grow horizontallySaturday, October 15, 11 23
  26. 26. DEMO bashoSaturday, October 15, 11 24
  27. 27. PUT curl -X PUT -H "x-riak-index-service_bin: news" -H "x-riak-index-range_int: 201110151300" -H "Content-Type: application/json" -d {"operation":"W", "key":"news/photo/20111015_small.jpg", "service":"news","range":"201110151300"} http://127.0.0.1:8098/riak/news/log_001 KEY VALUE For Secondary IndexSaturday, October 15, 11 25
  28. 28. GET curl http://127.0.0.1:8098/demo/news/log_001 | python -mjson.tool { "key": "news/photo/20111015_small.jpg", "operation": "W", "range": "201110151300", "service": "news" } KEY VALUE For Secondary IndexSaturday, October 15, 11 26
  29. 29. Seconday Index (2I) curl http://127.0.0.1:8098/buckets/demo/index/service_bin/news | python -mjson.tool { "keys": [ "log_001", "log_002" ] } KEY VALUE For Secondary IndexSaturday, October 15, 11 27
  30. 30. Map Reduce curl -H "Content-Type: application/json" http://127.0.0.1:8098/mapred --data @-<<EOF {"inputs":{"bucket":"demo","index":"range_int","key":"201110151300"}, "query":[ {"map": {"name":"Riak.mapValuesJson","language":"javascript","keep":false}}, {"reduce":{"language":"javascript","source":" function(v) { var json_object = eval(v); var result = {}; for (i = 0; i < json_object.length; i++) { var key = json_object[i].service + _ + json_object[i].operation; var cur = result[key] == undefined ? 0 : result[key]; result[key] = cur + 1; } return result; } "}}]} Result: {"news_W":2,"social_W":1} EOFSaturday, October 15, 11 28
  31. 31. basho_benchSaturday, October 15, 11 29
  32. 32. basho_bench Depend on Erlang Able to be ‘Easy Operation’ Able to create ‘Pretty Chart’ for KVS and OthersSaturday, October 15, 11 30
  33. 33. Benchmartk ResultSaturday, October 15, 11 31
  34. 34. basho_bench config {mode, max}. {duration, 15}. {concurrent, 5}. {driver, basho_bench_driver_riakclient}. {code_paths, ["deps/stats", "/Users/jmeredith/basho/riak/apps/riak_kv", "/Users/jmeredith/basho/riak/apps/riak_core"]}. {key_generator, {int_to_bin, {uniform_int, 35000}}}. {value_generator, {fixed_bin, 10000}}. {riakclient_nodes, [riak@127.0.0.1]}. {riakclient_mynode, [riak_bench@127.0.0.1, longnames]}. {riakclient_replies, 1}.Saturday, October 15, 11 32
  35. 35. Thank you for listeningSaturday, October 15, 11 33
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×