Your SlideShare is downloading. ×
0
Introduction to Riak
Introduction to Riak
Introduction to Riak
Introduction to Riak
Introduction to Riak
Introduction to Riak
Introduction to Riak
Introduction to Riak
Introduction to Riak
Introduction to Riak
Introduction to Riak
Introduction to Riak
Introduction to Riak
Introduction to Riak
Introduction to Riak
Introduction to Riak
Introduction to Riak
Introduction to Riak
Introduction to Riak
Introduction to Riak
Introduction to Riak
Introduction to Riak
Introduction to Riak
Introduction to Riak
Introduction to Riak
Introduction to Riak
Introduction to Riak
Introduction to Riak
Introduction to Riak
Introduction to Riak
Introduction to Riak
Introduction to Riak
Introduction to Riak
Introduction to Riak
Introduction to Riak
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Introduction to Riak

3,511

Published on

Published in: Technology, News & Politics
0 Comments
8 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,511
On Slideshare
0
From Embeds
0
Number of Embeds
8
Actions
Shares
0
Downloads
83
Comments
0
Likes
8
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Introduction to Riak Python hack-a-thon 2011.10.15 Yosuke Hara {twiter_id, yosukehara} Software DeveloperSaturday, October 15, 11 1
  • 2. Who am I ?Saturday, October 15, 11 2
  • 3. twitter-id: yosukehara RIA Distributed StorageSaturday, October 15, 11 3
  • 4. What’s ‘Riak’ ? bashoSaturday, October 15, 11 4
  • 5. KVS + ExtraSaturday, October 15, 11 5
  • 6. Basho Bitcask or LevelDB (Google, from 1.0)Saturday, October 15, 11 6
  • 7. Architecture Erlang/OTP Runtime Client APIs HTTP Protocol Buffers Erlang local client Request Coordination get put delete map-reduce consistent hashing handoff gossip membership node-liveness buckets Riak Core vnode master vnodes Workers storage backend Riak KVSaturday, October 15, 11 7
  • 8. The horizontal scaling bitsSaturday, October 15, 11 8
  • 9. Inspired Amazon’s Dynamo (2007) KVS (Key-Value Storage) + Extra Masterless, P2P-Replication Consistent Hashing Failover - Quorums, Hinted handoff Eventual ConsistencySaturday, October 15, 11 9
  • 10. Inspired Amazon’s Dynamo http://www.allthingsdistributed.com/2007/10/amazons_dynamo.html “The reliability and scalability of a system is dependent on how its application state is managed.” - Werner VogelsSaturday, October 15, 11 10
  • 11. KVS (Key-Value-Store) Simple operations - get, put, delete Value is mostly opaque (some metadata) Extras MapReduce (depends on Javascript) Links Full-text search (Optional - Riak Search) Secondary Indexes (from ver 1.0)Saturday, October 15, 11 11
  • 12. P2P - MasterlessSaturday, October 15, 11 12
  • 13. P2P - Masterless NOT SPOFSaturday, October 15, 11 12
  • 14. Consistent Hashing Virtual Node ID = SHA1( $KEY ) Virtual Node “Hash Ring” 2^160 / 32 Node-0 Node-1 Node-2 Node-3Saturday, October 15, 11 13
  • 15. Consistent Hashing Virtual Node ID = SHA1( $KEY ) Virtual Node Logical subdivision of the cluster “Hash Ring” 2^160 / 32 Node-0 Node-1 Node-2 Node-3Saturday, October 15, 11 13
  • 16. For Parallelism # of VNodes == Maximum Concurrent Reqs For Rebalancing the Cluster Smallest block that can be shifted to a new node For Resiliance The system restarts failed VNodesSaturday, October 15, 11 14
  • 17. Quorum Every request contacts all replicas of key N : number of replicas (default 3) R : read quorum W : write quorum Able to adjust to “Consistency Level”Saturday, October 15, 11 15
  • 18. Distributed Data (1) PUT File (3) if # of replicas needed for a successful Write > 2 # of replicas = 3 (4) return “ok” # of successful WRITEs = 2 Node-0 Node-1 Node-2 Node-3 (2) Replicate - Router on each nodeSaturday, October 15, 11 16
  • 19. Hinted Handoff X (1) PUT File X (3) if # of replicas needed X for a successful Write > 2 # of replicas = 3 X (4) return “ok” X # of successful WRITEs = 2 X Node-0 Node-1 X X Node-2 (2) Replicate Node-3 X (5) RecoverSaturday, October 15, 11 17
  • 20. Eventual ConsistencySaturday, October 15, 11 18
  • 21. CAP Theorem Consistency Availability Partition ToleranceSaturday, October 15, 11 19
  • 22. The doc-database bitsSaturday, October 15, 11 20
  • 23. Store objects as JSON or any format Link between objects No SQL - Query with Javascript Map-ReduceSaturday, October 15, 11 21
  • 24. The operations-friendly bitsSaturday, October 15, 11 22
  • 25. Web-shaped storage Store data in its original format Using HTTP Load balancing Caches No special-nodes - grow horizontallySaturday, October 15, 11 23
  • 26. DEMO bashoSaturday, October 15, 11 24
  • 27. PUT curl -X PUT -H "x-riak-index-service_bin: news" -H "x-riak-index-range_int: 201110151300" -H "Content-Type: application/json" -d {"operation":"W", "key":"news/photo/20111015_small.jpg", "service":"news","range":"201110151300"} http://127.0.0.1:8098/riak/news/log_001 KEY VALUE For Secondary IndexSaturday, October 15, 11 25
  • 28. GET curl http://127.0.0.1:8098/demo/news/log_001 | python -mjson.tool { "key": "news/photo/20111015_small.jpg", "operation": "W", "range": "201110151300", "service": "news" } KEY VALUE For Secondary IndexSaturday, October 15, 11 26
  • 29. Seconday Index (2I) curl http://127.0.0.1:8098/buckets/demo/index/service_bin/news | python -mjson.tool { "keys": [ "log_001", "log_002" ] } KEY VALUE For Secondary IndexSaturday, October 15, 11 27
  • 30. Map Reduce curl -H "Content-Type: application/json" http://127.0.0.1:8098/mapred --data @-<<EOF {"inputs":{"bucket":"demo","index":"range_int","key":"201110151300"}, "query":[ {"map": {"name":"Riak.mapValuesJson","language":"javascript","keep":false}}, {"reduce":{"language":"javascript","source":" function(v) { var json_object = eval(v); var result = {}; for (i = 0; i < json_object.length; i++) { var key = json_object[i].service + _ + json_object[i].operation; var cur = result[key] == undefined ? 0 : result[key]; result[key] = cur + 1; } return result; } "}}]} Result: {"news_W":2,"social_W":1} EOFSaturday, October 15, 11 28
  • 31. basho_benchSaturday, October 15, 11 29
  • 32. basho_bench Depend on Erlang Able to be ‘Easy Operation’ Able to create ‘Pretty Chart’ for KVS and OthersSaturday, October 15, 11 30
  • 33. Benchmartk ResultSaturday, October 15, 11 31
  • 34. basho_bench config {mode, max}. {duration, 15}. {concurrent, 5}. {driver, basho_bench_driver_riakclient}. {code_paths, ["deps/stats", "/Users/jmeredith/basho/riak/apps/riak_kv", "/Users/jmeredith/basho/riak/apps/riak_core"]}. {key_generator, {int_to_bin, {uniform_int, 35000}}}. {value_generator, {fixed_bin, 10000}}. {riakclient_nodes, [riak@127.0.0.1]}. {riakclient_mynode, [riak_bench@127.0.0.1, longnames]}. {riakclient_replies, 1}.Saturday, October 15, 11 32
  • 35. Thank you for listeningSaturday, October 15, 11 33

×