Riak at Engine Yard Cloud
Upcoming SlideShare
Loading in...5
×
 

Riak at Engine Yard Cloud

on

  • 2,208 views

Given at Big Nerd Ranch in Atlanta.

Given at Big Nerd Ranch in Atlanta.

Statistics

Views

Total Views
2,208
Views on SlideShare
2,079
Embed Views
129

Actions

Likes
0
Downloads
2
Comments
0

4 Embeds 129

http://inessombra.tumblr.com 90
http://assets.txmblr.com 33
https://twitter.com 5
https://assets.txmblr.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Riak at Engine Yard Cloud Riak at Engine Yard Cloud Presentation Transcript

  • Getting Started with Riak in the Cloud Big Nerd Ranch
  • About Me InesSombra@RandomMood
  • Our Agenda About Us Our Customers The Problem The Solution
  • Engine Yard
  • About US Clouds Regions100% CloudBased Stacks SupportCritical services DRMUST be up HA24/7/365
  • Our Customershttp://www.flickr.com/photos/meetrajesh/6972883401
  • An APPLICATIONAppApp DB Master us-east-1a DB ReplicaApp us-east-1b
  • Then AWS Crashed http://www.flickr.com/photos/brandongrasley/8227882239/
  • Our App AFTERAppApp XDB Master us-east-1a DB ReplicaApp us-east-1b
  • This sucked, A LOT! Time to strengthen our HA/DR Story$ Customers and us need to be up all the time!
  • Our ResearchData Models Consistency PartitionsRelational Are ACID Consistency propertiesDocument Availability reallyKey/Value needed?Column
  • Impedance Mismatch Relational Model
  • Impedance MismatchMemory data-structure
  • We needed a faulttolerant and highlyavailable (clustered) datastore
  • OSS, Dynamo-based,distributed K/V store Hello RIAK! Linearly scalable Fault-tolerant Finally provide HA/DR solution
  • Riak’s Use CasesApplication Type Key ValueSession User/Session id DataAdvertising Campaign id DataLogs Date Log file text, json,Content title, integer xml, media..
  • When is Riak a good fitWhen availability is moreimportant than consistencyWhen data can bemodeled as keys/valuesWhen the problem fits
  • Riak is OPS FRIENDLY X node 1.2.1 Growth node node 1.2.1 1.2.1 Reduction node 1.2.1 node 1.2.1 Upgrades node 1.3.0 Failures
  • World’s most famous slide
  • Riak is SO MUCH MORE Consistent Hashing Hinted Handoff Active Anti Entropy Tuneable consistency N|W|R
  • OUR APP AFTER App node 1.3.0 App node node 1.3.0 1.3.0 node node App 1.3.0 1.3.0
  • We need a RiakCluster but we have no hardware
  • RIAK In the Wild 18th and Harrison, San Francisco, CA
  • Let’s build A PRODUCT http://www.flickr.com/photos/worldbank/8261699153/
  • Your CHOICES You are trying to create a Riak cluster. Do you need any help? 1 Number of Nodes 2 Instance Type 3 Data Location 4 Backend Type
  • What we GIVE YOU
  • Our SUPPORT X2
  • HOWTO What youOPS to know
  • Prepare Yourself More instances to compensate for theperformance variability of virtualized resources
  • Choosing INSTANCES ^ m1.large & m1.xlarge EBS-optimized or PIOPs High I/O Quadruple XL (hi1.4xlarge) - SSDs!
  • Cluster Sizes 5 nodes or more Best performance & growth Scales linearly Don’t be stingy
  • Ring SIZE Change Default CANNOT BE RESET Ring Size at 256 (8-64 vnodes per node) DONT GO OVER 512
  • Our PROVISIONING Riak Cluster Riak Cluster App MyRiakCluster: node-0 haproxy HTTP or MyRiakCluster: node-1 Protobuff App haproxy MyRiakCluster: node-2 App MyRiakCluster: node-3 haproxy MyRiakCluster: node-4
  • Disable SWAP ^ Riak process pages should not hit swap Swap usage can result in unresponsive server Let the kernel kill it
  • Mount & SCHEDULER ^ Mount with noatime flag Use deadline scheduler on EBS cat /sys/block/xvdj1/queue/scheduler noop [deadline] cfq echo deadline > /sys/block/xvdj1/queue/ scheduler
  • Linux FILESYSTEMS ^ xfs, ext3 ext4 barrier = 0 data = writeback ZFS not recommended
  • Backup STRATEGY ^ Backup both the ring and data directories scp or rsync data dir EBS snapshots
  • Monitoring STRATEGY Stats Interface riak-admin status Collectd Nagios
  • Scaling RIAK^ Vertical riak-admin cluster replace <old> <new>^ Horizontal riak-admin cluster add <new>
  • How we SCALE RIAK
  • Hybrid Solutions FTWRIAK + PostgreSQLRIAK + FTS Engine*RIAK + Redis
  • Try OUR RIAK
  • How difficult IS IT?
  • Finally! A fewthoughts
  • Polyglot PERSISTENCE Pick the right tool for the job Apps & Services can have multiple databases
  • On SOA My super nice e-commerce siteShopping Cart & Completed Inventory and Recommendations Session Data Orders Item Pricing EngineSession Order Inventory & Nodes andStorage Persistence Price service relationsservice service service RDBMSK/V Store Document Graph Store Store NoSQL Distilled: Fowler & Sadalage
  • We RECOMMEND Don’t optimize too quickly Take time to understand your problem Complexity increases with more databases
  • Do KEEP IN MIND Different paradigms are different Document Relational != Store
  • WhateverGive meMOAR!
  • Great RESOURCES Little Riak Book (https://github.com/coderoshi/little_riak_book/) docs.basho.com blog.engineyard.com nosql.mypopescu.com
  • Questions?