Your SlideShare is downloading. ×
0
Ricon/West 2013: Adventures with Riak Pipe
Ricon/West 2013: Adventures with Riak Pipe
Ricon/West 2013: Adventures with Riak Pipe
Ricon/West 2013: Adventures with Riak Pipe
Ricon/West 2013: Adventures with Riak Pipe
Ricon/West 2013: Adventures with Riak Pipe
Ricon/West 2013: Adventures with Riak Pipe
Ricon/West 2013: Adventures with Riak Pipe
Ricon/West 2013: Adventures with Riak Pipe
Ricon/West 2013: Adventures with Riak Pipe
Ricon/West 2013: Adventures with Riak Pipe
Ricon/West 2013: Adventures with Riak Pipe
Ricon/West 2013: Adventures with Riak Pipe
Ricon/West 2013: Adventures with Riak Pipe
Ricon/West 2013: Adventures with Riak Pipe
Ricon/West 2013: Adventures with Riak Pipe
Ricon/West 2013: Adventures with Riak Pipe
Ricon/West 2013: Adventures with Riak Pipe
Ricon/West 2013: Adventures with Riak Pipe
Ricon/West 2013: Adventures with Riak Pipe
Ricon/West 2013: Adventures with Riak Pipe
Ricon/West 2013: Adventures with Riak Pipe
Ricon/West 2013: Adventures with Riak Pipe
Ricon/West 2013: Adventures with Riak Pipe
Ricon/West 2013: Adventures with Riak Pipe
Ricon/West 2013: Adventures with Riak Pipe
Ricon/West 2013: Adventures with Riak Pipe
Ricon/West 2013: Adventures with Riak Pipe
Ricon/West 2013: Adventures with Riak Pipe
Ricon/West 2013: Adventures with Riak Pipe
Ricon/West 2013: Adventures with Riak Pipe
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Ricon/West 2013: Adventures with Riak Pipe

1,020

Published on

Also make sure you check out Riak Pipe talks by Bryan Fink.

Also make sure you check out Riak Pipe talks by Bryan Fink.

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,020
On Slideshare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
12
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. DISTRIBUTING WORK ACROSS CLUSTERS Adventures With Riak Pipe Last Updated: October 31, 2013 Susan Potter @SusanPotter
  • 2. Background Concepts & Intuitions Applications WHOAMI 2
  • 3. 2013-10-31 . . Distributing Work Across Clusters WHOAMI whoami . . An app + middleware dev who has felt Ops pain too
  • 4. BACKGROUND
  • 5. Background Concepts & Intuitions Applications MODEL/ASSUMPTIONS 4
  • 6. Background Concepts & Intuitions Applications MOTIVATING USES 5
  • 7. Background Concepts & Intuitions Applications COMMON APPROACHES 6
  • 8. Background Concepts & Intuitions Applications WHY DISTRIBUTE? WHY DECENTRALIZE? Probability of system failure → Large datasets (data locality) → Eliminate SPOFs → Commodity hardware → 7
  • 9. Background Concepts & Intuitions Applications 8
  • 10. CONCEPTS & INTUITIONS
  • 11. Background Concepts & Intuitions Applications LAYERS 10
  • 12. Background Concepts & Intuitions Applications TERMINOLOGY (CORE) → Partition → Vnode → Hinted Handoff → Read Repair 11
  • 13. Background Concepts & Intuitions Applications TERMINOLOGY (PIPE) → Pipe → Fitting ("phase") → Queue → Worker 12
  • 14. Background Concepts & Intuitions Applications EXAMPLE UNIX PIPE 1 2 3 find . -name "*. rb" | xargs egrep "#.*? TODO:" | wc -l Character-based, through file descriptors 13
  • 15. Background Concepts & Intuitions Applications EXAMPLE FUNCTION COMPOSITION 1 ( length . mapToUpper . sanitize ) input Value based, through functions 14
  • 16. Background Concepts & Intuitions Applications EXAMPLE RIAK PIPE 1 2 3 4 5 6 7 [ # fitting_spec { , , , # fitting_spec { , , ] name= fetch_trades module = riskmgr_fetch_trades ...} name= calc_var module = riskmgr_calc_var ...} Message-based, across nodes 15
  • 17. Background Concepts & Intuitions Applications VNODE AND WORKERS → vnode manages lifecycle and queues of workers → vnode worker processes inputs 16
  • 18. Background Concepts & Intuitions Applications VNODE WORKER BEHAVIOR 1 2 3 4 5 6 7 behaviour_info ( callbacks ) -> [{init ,2}, {process ,3}, {done ,1}]; behaviour_info ( _Other ) -> undefined . %% Optionally two more too , not required . 17
  • 19. Background Concepts & Intuitions Applications DISTRIBUTION: CHASHFUN → Use random uniform hash fun to saturate workers → See Bryan Fink's previous Pipe presentations 18
  • 20. Background Concepts & Intuitions Applications VALIDATION 1 2 3 4 5 % # fitting_spec { module = ... , arg = 43, ... } % validate_arg /1 validate_arg (Arg) when is_integer (Arg) -> ok; validate_arg (Arg) when is_atom (Arg) -> ok; validate_arg (_) -> {error , " Argument must be a valid integer or atom"}. 19
  • 21. Background Concepts & Intuitions Applications LEAKY PIPES 20
  • 22. Background Concepts & Intuitions Applications RIAK (CORE) MECHANICS → Handoff Migrate vnodes from node to node 1 2 3 % Old Node % Called with last known State of worker archive /1 % returns {ok , Archive } 4 5 6 7 % New Node % Called with Archive and State from old node handoff /2 % returns {ok , NewState } 21
  • 23. Background Concepts & Intuitions Applications FAILURES → validate_arg Seen above → Pipe client process dies YOLO → nval How many vnodes to ask before failing, default 1 → vnode worker logic Think using dataflow semantics 22
  • 24. Background Concepts & Intuitions Applications EXISTING FITTINGS → riak_pipe_w_tee Useful for intermediate results → riak_pipe_w_xform Simple delegater: 3-arity function → riak_pipe_w_reduce What you would expect: simple accumulating reduce → riak_kv_pipe_get Take advantage of data locality with cohosted KV store 23
  • 25. APPLICATIONS
  • 26. Background Concepts & Intuitions Applications KNOWN USES Riak's Map/Reduce → Risk metrics → Tenant Usage → 25
  • 27. Background Concepts & Intuitions Applications TROUBLESHOOTING → riak_pipe:status/1 Provides fittings, processed, failures, queue_length, etc stats → riak_pipe_w_crash fitting Used to test Riak Pipe → riak_pipe:active_pipelines/1 See all active pipelines → riak_pipe_cinfo module cluster info interrogation module 26
  • 28. Background Concepts & Intuitions Applications FURTHER WORK → More applications (e.g. genetics, 3rd party APIs with rate limits) → Decentralized pipe control → Measure "completeness" of resultset 27
  • 29. Background Concepts & Intuitions Applications RELATED WORK → "Pipe" libraries EVERYWHERE (pipe, scalaz-stream) → riak_pg → Map-Reduce frameworks → Staged Event Driven Architecture → Event Stream Processing → Dataflow multi-stage processing 28
  • 30. Background Concepts & Intuitions Applications ROYAL FAIL http://www.flickr.com/photos/dadavidov/ 29
  • 31. Background Concepts & Intuitions Applications QUESTIONS Questions? 30

×