Riak at Engine Yard Cloud

2,578 views
2,460 views

Published on

Given at Big Nerd Ranch in Atlanta.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,578
On SlideShare
0
From Embeds
0
Number of Embeds
151
Actions
Shares
0
Downloads
5
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Riak at Engine Yard Cloud

  1. 1. Getting Started with Riak in the Cloud Big Nerd Ranch
  2. 2. About Me InesSombra@RandomMood
  3. 3. Our Agenda About Us Our Customers The Problem The Solution
  4. 4. Engine Yard
  5. 5. About US Clouds Regions100% CloudBased Stacks SupportCritical services DRMUST be up HA24/7/365
  6. 6. Our Customershttp://www.flickr.com/photos/meetrajesh/6972883401
  7. 7. An APPLICATIONAppApp DB Master us-east-1a DB ReplicaApp us-east-1b
  8. 8. Then AWS Crashed http://www.flickr.com/photos/brandongrasley/8227882239/
  9. 9. Our App AFTERAppApp XDB Master us-east-1a DB ReplicaApp us-east-1b
  10. 10. This sucked, A LOT! Time to strengthen our HA/DR Story$ Customers and us need to be up all the time!
  11. 11. Our ResearchData Models Consistency PartitionsRelational Are ACID Consistency propertiesDocument Availability reallyKey/Value needed?Column
  12. 12. Impedance Mismatch Relational Model
  13. 13. Impedance MismatchMemory data-structure
  14. 14. We needed a faulttolerant and highlyavailable (clustered) datastore
  15. 15. OSS, Dynamo-based,distributed K/V store Hello RIAK! Linearly scalable Fault-tolerant Finally provide HA/DR solution
  16. 16. Riak’s Use CasesApplication Type Key ValueSession User/Session id DataAdvertising Campaign id DataLogs Date Log file text, json,Content title, integer xml, media..
  17. 17. When is Riak a good fitWhen availability is moreimportant than consistencyWhen data can bemodeled as keys/valuesWhen the problem fits
  18. 18. Riak is OPS FRIENDLY X node 1.2.1 Growth node node 1.2.1 1.2.1 Reduction node 1.2.1 node 1.2.1 Upgrades node 1.3.0 Failures
  19. 19. World’s most famous slide
  20. 20. Riak is SO MUCH MORE Consistent Hashing Hinted Handoff Active Anti Entropy Tuneable consistency N|W|R
  21. 21. OUR APP AFTER App node 1.3.0 App node node 1.3.0 1.3.0 node node App 1.3.0 1.3.0
  22. 22. We need a RiakCluster but we have no hardware
  23. 23. RIAK In the Wild 18th and Harrison, San Francisco, CA
  24. 24. Let’s build A PRODUCT http://www.flickr.com/photos/worldbank/8261699153/
  25. 25. Your CHOICES You are trying to create a Riak cluster. Do you need any help? 1 Number of Nodes 2 Instance Type 3 Data Location 4 Backend Type
  26. 26. What we GIVE YOU
  27. 27. Our SUPPORT X2
  28. 28. HOWTO What youOPS to know
  29. 29. Prepare Yourself More instances to compensate for theperformance variability of virtualized resources
  30. 30. Choosing INSTANCES ^ m1.large & m1.xlarge EBS-optimized or PIOPs High I/O Quadruple XL (hi1.4xlarge) - SSDs!
  31. 31. Cluster Sizes 5 nodes or more Best performance & growth Scales linearly Don’t be stingy
  32. 32. Ring SIZE Change Default CANNOT BE RESET Ring Size at 256 (8-64 vnodes per node) DONT GO OVER 512
  33. 33. Our PROVISIONING Riak Cluster Riak Cluster App MyRiakCluster: node-0 haproxy HTTP or MyRiakCluster: node-1 Protobuff App haproxy MyRiakCluster: node-2 App MyRiakCluster: node-3 haproxy MyRiakCluster: node-4
  34. 34. Disable SWAP ^ Riak process pages should not hit swap Swap usage can result in unresponsive server Let the kernel kill it
  35. 35. Mount & SCHEDULER ^ Mount with noatime flag Use deadline scheduler on EBS cat /sys/block/xvdj1/queue/scheduler noop [deadline] cfq echo deadline > /sys/block/xvdj1/queue/ scheduler
  36. 36. Linux FILESYSTEMS ^ xfs, ext3 ext4 barrier = 0 data = writeback ZFS not recommended
  37. 37. Backup STRATEGY ^ Backup both the ring and data directories scp or rsync data dir EBS snapshots
  38. 38. Monitoring STRATEGY Stats Interface riak-admin status Collectd Nagios
  39. 39. Scaling RIAK^ Vertical riak-admin cluster replace <old> <new>^ Horizontal riak-admin cluster add <new>
  40. 40. How we SCALE RIAK
  41. 41. Hybrid Solutions FTWRIAK + PostgreSQLRIAK + FTS Engine*RIAK + Redis
  42. 42. Try OUR RIAK
  43. 43. How difficult IS IT?
  44. 44. Finally! A fewthoughts
  45. 45. Polyglot PERSISTENCE Pick the right tool for the job Apps & Services can have multiple databases
  46. 46. On SOA My super nice e-commerce siteShopping Cart & Completed Inventory and Recommendations Session Data Orders Item Pricing EngineSession Order Inventory & Nodes andStorage Persistence Price service relationsservice service service RDBMSK/V Store Document Graph Store Store NoSQL Distilled: Fowler & Sadalage
  47. 47. We RECOMMEND Don’t optimize too quickly Take time to understand your problem Complexity increases with more databases
  48. 48. Do KEEP IN MIND Different paradigms are different Document Relational != Store
  49. 49. WhateverGive meMOAR!
  50. 50. Great RESOURCES Little Riak Book (https://github.com/coderoshi/little_riak_book/) docs.basho.com blog.engineyard.com nosql.mypopescu.com
  51. 51. Questions?

×