Introduction to Riak
Upcoming SlideShare
Loading in...5
×
 

Introduction to Riak

on

  • 3,792 views

 

Statistics

Views

Total Views
3,792
Views on SlideShare
3,144
Embed Views
648

Actions

Likes
8
Downloads
73
Comments
0

15 Embeds 648

http://yosuke-perspective.blogspot.jp 319
http://yosuke-perspective.blogspot.com 297
http://yosuke-perspective.blogspot.ru 13
http://a0.twimg.com 3
http://yosuke-perspective.blogspot.sg 3
http://webcache.googleusercontent.com 2
http://yosuke-perspective.blogspot.de 2
http://yosuke-perspective.blogspot.ca 2
http://yosuke-perspective.blogspot.fr 1
http://yosuke-perspective.blogspot.hk 1
http://yosuke-perspective.blogspot.ie 1
http://yosuke-perspective.blogspot.mx 1
http://yosuke-perspective.blogspot.com.es 1
http://paper.li 1
http://leapf.org 1
More...

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Introduction to Riak Presentation Transcript

  • 1. Introduction to Riak Python hack-a-thon 2011.10.15 Yosuke Hara {twiter_id, yosukehara} Software DeveloperSaturday, October 15, 11 1
  • 2. Who am I ?Saturday, October 15, 11 2
  • 3. twitter-id: yosukehara RIA Distributed StorageSaturday, October 15, 11 3
  • 4. What’s ‘Riak’ ? bashoSaturday, October 15, 11 4
  • 5. KVS + ExtraSaturday, October 15, 11 5
  • 6. Basho Bitcask or LevelDB (Google, from 1.0)Saturday, October 15, 11 6
  • 7. Architecture Erlang/OTP Runtime Client APIs HTTP Protocol Buffers Erlang local client Request Coordination get put delete map-reduce consistent hashing handoff gossip membership node-liveness buckets Riak Core vnode master vnodes Workers storage backend Riak KVSaturday, October 15, 11 7
  • 8. The horizontal scaling bitsSaturday, October 15, 11 8
  • 9. Inspired Amazon’s Dynamo (2007) KVS (Key-Value Storage) + Extra Masterless, P2P-Replication Consistent Hashing Failover - Quorums, Hinted handoff Eventual ConsistencySaturday, October 15, 11 9
  • 10. Inspired Amazon’s Dynamo http://www.allthingsdistributed.com/2007/10/amazons_dynamo.html “The reliability and scalability of a system is dependent on how its application state is managed.” - Werner VogelsSaturday, October 15, 11 10
  • 11. KVS (Key-Value-Store) Simple operations - get, put, delete Value is mostly opaque (some metadata) Extras MapReduce (depends on Javascript) Links Full-text search (Optional - Riak Search) Secondary Indexes (from ver 1.0)Saturday, October 15, 11 11
  • 12. P2P - MasterlessSaturday, October 15, 11 12
  • 13. P2P - Masterless NOT SPOFSaturday, October 15, 11 12
  • 14. Consistent Hashing Virtual Node ID = SHA1( $KEY ) Virtual Node “Hash Ring” 2^160 / 32 Node-0 Node-1 Node-2 Node-3Saturday, October 15, 11 13
  • 15. Consistent Hashing Virtual Node ID = SHA1( $KEY ) Virtual Node Logical subdivision of the cluster “Hash Ring” 2^160 / 32 Node-0 Node-1 Node-2 Node-3Saturday, October 15, 11 13
  • 16. For Parallelism # of VNodes == Maximum Concurrent Reqs For Rebalancing the Cluster Smallest block that can be shifted to a new node For Resiliance The system restarts failed VNodesSaturday, October 15, 11 14
  • 17. Quorum Every request contacts all replicas of key N : number of replicas (default 3) R : read quorum W : write quorum Able to adjust to “Consistency Level”Saturday, October 15, 11 15
  • 18. Distributed Data (1) PUT File (3) if # of replicas needed for a successful Write > 2 # of replicas = 3 (4) return “ok” # of successful WRITEs = 2 Node-0 Node-1 Node-2 Node-3 (2) Replicate - Router on each nodeSaturday, October 15, 11 16
  • 19. Hinted Handoff X (1) PUT File X (3) if # of replicas needed X for a successful Write > 2 # of replicas = 3 X (4) return “ok” X # of successful WRITEs = 2 X Node-0 Node-1 X X Node-2 (2) Replicate Node-3 X (5) RecoverSaturday, October 15, 11 17
  • 20. Eventual ConsistencySaturday, October 15, 11 18
  • 21. CAP Theorem Consistency Availability Partition ToleranceSaturday, October 15, 11 19
  • 22. The doc-database bitsSaturday, October 15, 11 20
  • 23. Store objects as JSON or any format Link between objects No SQL - Query with Javascript Map-ReduceSaturday, October 15, 11 21
  • 24. The operations-friendly bitsSaturday, October 15, 11 22
  • 25. Web-shaped storage Store data in its original format Using HTTP Load balancing Caches No special-nodes - grow horizontallySaturday, October 15, 11 23
  • 26. DEMO bashoSaturday, October 15, 11 24
  • 27. PUT curl -X PUT -H "x-riak-index-service_bin: news" -H "x-riak-index-range_int: 201110151300" -H "Content-Type: application/json" -d {"operation":"W", "key":"news/photo/20111015_small.jpg", "service":"news","range":"201110151300"} http://127.0.0.1:8098/riak/news/log_001 KEY VALUE For Secondary IndexSaturday, October 15, 11 25
  • 28. GET curl http://127.0.0.1:8098/demo/news/log_001 | python -mjson.tool { "key": "news/photo/20111015_small.jpg", "operation": "W", "range": "201110151300", "service": "news" } KEY VALUE For Secondary IndexSaturday, October 15, 11 26
  • 29. Seconday Index (2I) curl http://127.0.0.1:8098/buckets/demo/index/service_bin/news | python -mjson.tool { "keys": [ "log_001", "log_002" ] } KEY VALUE For Secondary IndexSaturday, October 15, 11 27
  • 30. Map Reduce curl -H "Content-Type: application/json" http://127.0.0.1:8098/mapred --data @-<<EOF {"inputs":{"bucket":"demo","index":"range_int","key":"201110151300"}, "query":[ {"map": {"name":"Riak.mapValuesJson","language":"javascript","keep":false}}, {"reduce":{"language":"javascript","source":" function(v) { var json_object = eval(v); var result = {}; for (i = 0; i < json_object.length; i++) { var key = json_object[i].service + _ + json_object[i].operation; var cur = result[key] == undefined ? 0 : result[key]; result[key] = cur + 1; } return result; } "}}]} Result: {"news_W":2,"social_W":1} EOFSaturday, October 15, 11 28
  • 31. basho_benchSaturday, October 15, 11 29
  • 32. basho_bench Depend on Erlang Able to be ‘Easy Operation’ Able to create ‘Pretty Chart’ for KVS and OthersSaturday, October 15, 11 30
  • 33. Benchmartk ResultSaturday, October 15, 11 31
  • 34. basho_bench config {mode, max}. {duration, 15}. {concurrent, 5}. {driver, basho_bench_driver_riakclient}. {code_paths, ["deps/stats", "/Users/jmeredith/basho/riak/apps/riak_kv", "/Users/jmeredith/basho/riak/apps/riak_core"]}. {key_generator, {int_to_bin, {uniform_int, 35000}}}. {value_generator, {fixed_bin, 10000}}. {riakclient_nodes, [riak@127.0.0.1]}. {riakclient_mynode, [riak_bench@127.0.0.1, longnames]}. {riakclient_replies, 1}.Saturday, October 15, 11 32
  • 35. Thank you for listeningSaturday, October 15, 11 33