Your SlideShare is downloading. ×
Getting started with Riak in the Cloud
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Getting started with Riak in the Cloud

596
views

Published on

Given at RICON 2012

Given at RICON 2012

Published in: Technology

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
596
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Getting started withRiak in the Cloud
  • 2. I work with I work with Code! Data!Michael Brodhead Ines Sombra @mojinations @RandomMood
  • 3. AgendaApplication layer Database layer
  • 4. A tale of 2 apps• One brand new (no code written)• One in production running MySQL
  • 5. MultipleRegions
  • 6. MultipleClouds
  • 7. 11
  • 8. The Librariesriak-client• simple Ruby access to Riaks HTTP & protocol buffer APIs
  • 9. The LibrariesRipple• Higher level, ORM interface• ActiveModel based buffer APIs
  • 10. Simplicity
  • 11. The Librariesdm-riak• Already using DataMapper• Hoping for trivial switchover
  • 12. The LibrariesRiak Shim• database.yml• Bucket naming• Hides JSON• 2i
  • 13. Challenges • Unit tests • Data migration • Deployment
  • 14. Testing
  • 15. Testing
  • 16. Deployment
  • 17. Use your app logic
  • 18. How do we get here?
  • 19. “We need aRiak cluster” (@mojinations circa 3 months ago) ... but we have no hardware
  • 20. Let’s do it!
  • 21. Why Riak Product?• Linearly Scalable• Fault-tolerant• Ops friendly• Product lacks HA options
  • 22. Riak @ Engine Yard • We’ll use it first • No custom Chef • Provisioning, configuration, monitoring, & backups
  • 23. Your Choices• Number of nodes• Instance type You are trying to create a Riak cluster. Do you need any help?• Data location• Backend type
  • 24. AWS OpsThe Riak on AWS slide
  • 25. Instances & I/O• Riak’s I/O patterns• Small blobs from many places on disk• EBS best at bulk reads &writes
  • 26. Instance types• m1.large & m1.xlarge• EBS-optimized NEW!• Consider PIOPS NEW!• High I/O Quadruple XL (hi1.4xlarge) - SSDs! NEW!
  • 27. Prepare YourselfMore instances to compensatefor the performance variabilityof virtualized resources
  • 28. Cluster Sizes• 5 nodes or more Best performance & growth• Scales linearly• Don’t be stingy
  • 29. Ring Size• Change default value!• Cannot be reset• Ring size: 256 8/64 vnodes per node• Don’t go above 512
  • 30. Provisioning Riak Cluster App MyRiakCluster: node-0haproxy HTTP or MyRiakCluster: node-1 Protobuff Apphaproxy MyRiakCluster: node-2 App MyRiakCluster: node-3haproxy MyRiakCluster: node-4
  • 31. Disable Swap• Riak process pages should not hit swap• Swap usage can result in unresponsive server• Let the kernel kill it
  • 32. Mount & Scheduler• mount with noatime flag• Use deadline scheduler on EBS cat /sys/block/xvdj1/queue/scheduler noop [deadline] cfq echo deadline > /sys/block/xvdj1/ queue/scheduler
  • 33. Linux Filesystems• xfs, ext3• ext4 • barrier = 0 • data = writeback• ZFS not recommended
  • 34. Backend choices• Bitcask* • help mitigate EBS• LevelDB Can be changed• Memory and optimized!
  • 35. Kernel Optimizations• /etc/sysctl.conf vm.swappiness = 0 net.core.somaxconn= 4000 net.ipv4.tcp_timestamps = 0 net.ipv4.tcp_window_scaling = 1 net.ipv4.tcp_max_syn_backlog = 40000 net.ipv4.tcp_sack = 1 net.ipv4.tcp_fin_timeout = 15 net.ipv4.tcp_keepalive_intvl = 30 net.ipv4.tcp_tw_reuse = 1 http://wiki.basho.com/Linux-Performance-Tuning.html
  • 36. Monitoring• stats or riak-admin status• Collectd & Nagios• Riaknostic• Riak Control (maybe soon)
  • 37. Backups• Backup both the ring and data directories• Integrate with EY tools• scp or rsync data directory• EBS snapshots
  • 38. Scaling Riak• Verticalriak-admin cluster replace <old> <new>• Horizontal • reduces node load • batch is better
  • 39. Benchmarks
  • 40. Tests• 5-node cluster / 5 hours• 50 concurrent connections• gets 10 /puts 5 /deletes 1• Bitcask backend• Protobuffs protocol
  • 41. Medium - PIOPS 500
  • 42. Medium - PIOPS 1K
  • 43. Medium - EphemeralFAIL!
  • 44. XL - EBS
  • 45. XL -PIOPS 500
  • 46. 4XL - EBS
  • 47. 4XL - PIOPS 1k
  • 48. What’s next?