Real world capacity planning: Cassandra on blades and big iron   July 2011
About me <ul><li>Hadoop System Admin @ media6degrees </li></ul><ul><ul><li>Watch cassandra servers as well </li></ul></ul>...
Media6 Degrees  <ul><li>Social Targeting in online advertising </li></ul><ul><li>Real Time Bidding -  A dynamic auction pr...
What is this talk about?  <ul><li>Real World Capacity Planning </li></ul><ul><li>Been running c* in production > 1 year </...
Using puppet, chef... from day 1 <ul><li>“ I am going to chose Cassandra 0.6.0-beta-1 over 0.5.x so I am future proof” -- ...
Calculating Data size on disk <ul><li>SSTable format currently not compressed </li></ul><ul><li>Repairs, joins, and moves ...
Speed of disk <ul><li>The faster the better! </li></ul><ul><li>But faster + bigger gets expensive and challenging </li></u...
Disk formatting <ul><li>ext4 everywhere </li></ul><ul><li>Deletes are much better then ext3 </li></ul><ul><li>Noticeable p...
Memory <ul><li>Garbage collection is on a separate thread(s) </li></ul><ul><li>Each request creates temporary objects </li...
CPU <ul><li>Workload could be more disk then CPU bound  </li></ul><ul><li>High load needs a CPU to clean up java garbage <...
Different workloads <ul><li>Structured log format of C* has deep implications </li></ul><ul><li>Is data written once or do...
Large Disk / Big Iron key points <ul><li>RAID0 mean time to failure with bigger stripes </li></ul><ul><li>Java can not add...
Blade server key points <ul><li>Management software gives cloud computing vibe  </li></ul><ul><li>Cassandra internode traf...
Schema lessons <ul><li>You only need one column family. not always true </li></ul><ul><li>Infrequently read data in the sa...
Capacity Planning rule #1 Know your hard drive limits
Capacity Planning rule #2 Writes are fast, until c* flushes and  compacts so much, that they are not
Capacity Planning rule #3 <ul><li>Row cache is fools gold </li></ul><ul><ul><li>Faster then a read from disk cache </li></...
Capacity Planning rule #4 <ul><li>Do not upgrade tomorrow what  you can upgrade today </li></ul><ul><ul><li>Joining nodes ...
Capacity Planning rule #5 <ul><li>Know your traffic patterns better then yourself  </li></ul>
The use case: Dr. Real Time and Mr. Batch
Dr. Real Time <ul><ul><li>Real time bidding needs low latency </li></ul></ul><ul><ul><li>Peak traffic during the day </li>...
Dr. Real Time's Lab <ul><ul><li>Experiments with Xmx vs VFS caching </li></ul></ul><ul><ul><li>Experiments with cache sizi...
Mr. Batch <ul><ul><li>Night falls and users sleep </li></ul></ul><ul><ul><li>Batch/Back loading data (bulk inserts) </li><...
Mr. Batch rampaging  through the data <ul><ul><li>Bulk loading </li></ul></ul><ul><ul><ul><li>Write at quorum, c* work har...
Questions <ul><ul><li>???  </li></ul></ul>
Upcoming SlideShare
Loading in...5
×

Real world capacity

4,298

Published on

1 Comment
3 Likes
Statistics
Notes
  • Add it on the nosql.io. Thank you for the great presentation.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
4,298
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
114
Comments
1
Likes
3
Embeds 0
No embeds

No notes for slide
  • 10/01/10
  • Real world capacity

    1. 1. Real world capacity planning: Cassandra on blades and big iron July 2011
    2. 2. About me <ul><li>Hadoop System Admin @ media6degrees </li></ul><ul><ul><li>Watch cassandra servers as well </li></ul></ul><ul><ul><li>Write code (hadoop filecrusher) </li></ul></ul><ul><li>Hive Committer </li></ul><ul><ul><li>Variable substitution, UDFs like atan, rough draft of c* handler </li></ul></ul><ul><li>Epic Cassandra Contributor (not!) </li></ul><ul><ul><li>CLI should allow users to chose consistency level </li></ul></ul><ul><ul><li>NodeCmd should be able to view Compaction Statistics </li></ul></ul><ul><li>Self proclaimed president of Cassandra fan club </li></ul><ul><ul><li>Cassandra NYC User Group </li></ul></ul><ul><ul><li>High Performance Cassandra Cookbook </li></ul></ul>
    3. 3. Media6 Degrees <ul><li>Social Targeting in online advertising </li></ul><ul><li>Real Time Bidding - A dynamic auction process where each impression is bid for in (near) real time </li></ul><ul><li>Cassandra @ work storing: </li></ul><ul><ul><li>Visit Data </li></ul></ul><ul><ul><li>Ad History </li></ul></ul><ul><ul><li>Id Mapping </li></ul></ul><ul><li>Multiple Data Centers (home brew replication) </li></ul><ul><li>Back end tools hadoop (Data mining, bulk loads) </li></ul><ul><li>Front end tomcat, mysql + cassandra (lookup data) </li></ul>
    4. 4. What is this talk about? <ul><li>Real World Capacity Planning </li></ul><ul><li>Been running c* in production > 1 year </li></ul><ul><li>Started with a hand full of nodes also running tomcat and Replication Factor 2! </li></ul><ul><li>Grew data from 0-10 TB data </li></ul><ul><li>Grew from 0-751,398,530 reads / day </li></ul><ul><li>All types of fun along the way </li></ul>
    5. 5. Using puppet, chef... from day 1 <ul><li>“ I am going to chose Cassandra 0.6.0-beta-1 over 0.5.x so I am future proof” -- Famous quote by me </li></ul><ul><li>Cassandra is active </li></ul><ul><ul><li>new versions are coming </li></ul></ul><ul><ul><li>Rolling restarts between minors </li></ul></ul><ul><ul><li>But much better to get all to same rev quickly </li></ul></ul><ul><li>New nodes are coming do not let them: </li></ul><ul><ul><li>start with the wrong settings </li></ul></ul><ul><ul><li>fail because you forgot open file limits, etc </li></ul></ul>
    6. 6. Calculating Data size on disk <ul><li>SSTable format currently not compressed </li></ul><ul><li>Repairs, joins, and moves need “wiggle room” </li></ul><ul><li>Smaller keys and column names save space </li></ul><ul><li>Enough free space to compact your largest column family </li></ul><ul><li>Snapshots keep SSTables around after compaction </li></ul><ul><li>Most *Nix files systems need free space to avoid performance loss to fragmentation! </li></ul>
    7. 7. Speed of disk <ul><li>The faster the better! </li></ul><ul><li>But faster + bigger gets expensive and challenging </li></ul><ul><li>RAID0 </li></ul><ul><ul><li>Faster for streaming </li></ul></ul><ul><ul><li>not necessarily seeking </li></ul></ul><ul><ul><li>Fragile, larger the stripe, higher chance of failure </li></ul></ul><ul><li>RAID5 </li></ul><ul><ul><li>Not as fast but survives disk failure </li></ul></ul><ul><li>Battery backed cache helps but is $$$ </li></ul><ul><li>The dedicated commit log decision </li></ul>
    8. 8. Disk formatting <ul><li>ext4 everywhere </li></ul><ul><li>Deletes are much better then ext3 </li></ul><ul><li>Noticeable performance as disks get full </li></ul><ul><li>A full async mode for risk takers </li></ul><ul><li>Obligatory noatime fstab setting </li></ul><ul><li>using multiple file systems can result in multiple caches (check slabtop) </li></ul><ul><li>Mention XFS </li></ul>
    9. 9. Memory <ul><li>Garbage collection is on a separate thread(s) </li></ul><ul><li>Each request creates temporary objects </li></ul><ul><li>Cassandra's fast writes go to Memtables </li></ul><ul><ul><li>You will never guess what they use :) </li></ul></ul><ul><li>Bloom filter data is in memory </li></ul><ul><li>Key cache and Row cache </li></ul><ul><li>For low latency RAM must be some % of data </li></ul><ul><ul><li>RAM not used by process is OS cache </li></ul></ul>
    10. 10. CPU <ul><li>Workload could be more disk then CPU bound </li></ul><ul><li>High load needs a CPU to clean up java garbage </li></ul><ul><li>Other then serving requests, compaction uses resources </li></ul>
    11. 11. Different workloads <ul><li>Structured log format of C* has deep implications </li></ul><ul><li>Is data written once or does it change over time? </li></ul><ul><li>How high is data churn? </li></ul><ul><li>How random is the read/write pattern? </li></ul><ul><li>What is the write/read percentage? </li></ul><ul><li>What are your latency requirements? </li></ul>
    12. 12. Large Disk / Big Iron key points <ul><li>RAID0 mean time to failure with bigger stripes </li></ul><ul><li>Java can not address large heaps well </li></ul><ul><li>Compactions/Joins/repairs take a long time </li></ul><ul><ul><li>Lowers agility when joining a node could take hours </li></ul></ul><ul><li>Maintaining high RAM to Data percentage costly IE 2 machines with 32GB vs 1 machine with 64GB </li></ul><ul><li>Capacity heavily diminished with loss of one node </li></ul>
    13. 13. Blade server key points <ul><li>Management software gives cloud computing vibe </li></ul><ul><li>Cassandra internode traffic on blade back plane </li></ul><ul><li>Usually support 1-2 on board disk SCSI/SSD </li></ul><ul><li>Usually support RAM configurations up to 128G </li></ul><ul><li>Single and duel socket CPU </li></ul><ul><li>No exotic RAID options </li></ul>
    14. 14. Schema lessons <ul><li>You only need one column family. not always true </li></ul><ul><li>Infrequently read data in the same CF as frequently data compete for “cache” </li></ul><ul><li>Separating allows employing multiple cache options </li></ul><ul><li>Rows that are written or updated get fragmented </li></ul>
    15. 15. Capacity Planning rule #1 Know your hard drive limits
    16. 16. Capacity Planning rule #2 Writes are fast, until c* flushes and compacts so much, that they are not
    17. 17. Capacity Planning rule #3 <ul><li>Row cache is fools gold </li></ul><ul><ul><li>Faster then a read from disk cache </li></ul></ul><ul><ul><li>Memory use (row key + columns and values) </li></ul></ul><ul><ul><li>Causes memory pressure (data in and out of mem) </li></ul></ul><ul><ul><li>Fails with large rows </li></ul></ul><ul><ul><li>Cold on startup </li></ul></ul>
    18. 18. Capacity Planning rule #4 <ul><li>Do not upgrade tomorrow what you can upgrade today </li></ul><ul><ul><li>Joining nodes is intensive on the cluster </li></ul></ul><ul><ul><li>Do not wait till c* disks are 99% utilized </li></ul></ul><ul><ul><li>Do not get 100% benefit of new nodes until neighbors are cleaned </li></ul></ul><ul><ul><li>Doubling nodes results in less move steps </li></ul></ul><ul><ul><li>Adding RAM is fast and takes heat of hard disk </li></ul></ul>
    19. 19. Capacity Planning rule #5 <ul><li>Know your traffic patterns better then yourself </li></ul>
    20. 20. The use case: Dr. Real Time and Mr. Batch
    21. 21. Dr. Real Time <ul><ul><li>Real time bidding needs low latency </li></ul></ul><ul><ul><li>Peak traffic during the day </li></ul></ul><ul><ul><li>Need to keep a high cache hit rate </li></ul></ul><ul><ul><li>Avoid compact, repair, cleanup, joins </li></ul></ul>
    22. 22. Dr. Real Time's Lab <ul><ul><li>Experiments with Xmx vs VFS caching </li></ul></ul><ul><ul><li>Experiments with cache sizing </li></ul></ul><ul><ul><li>Studying graphs as new releases and features are added </li></ul></ul><ul><ul><li>Monitoring dropped messages, garbage collection </li></ul></ul><ul><ul><li>Dr. Real Time enjoys lots of memory for GB of data on disk </li></ul></ul><ul><ul><ul><li>Enjoys reading (data), writing as well </li></ul></ul></ul><ul><ul><ul><li>Nice sized memtables help to not pollute vfs cache </li></ul></ul></ul>
    23. 23. Mr. Batch <ul><ul><li>Night falls and users sleep </li></ul></ul><ul><ul><li>Batch/Back loading data (bulk inserts) </li></ul></ul><ul><ul><li>Finding and removing old data (range scanning) </li></ul></ul><ul><ul><li>Maintenance work (nodetool) </li></ul></ul>
    24. 24. Mr. Batch rampaging through the data <ul><ul><li>Bulk loading </li></ul></ul><ul><ul><ul><li>Write at quorum, c* work harder on front end </li></ul></ul></ul><ul><ul><ul><li>Turning off compaction </li></ul></ul></ul><ul><ul><ul><ul><li>For short burst fine, but we are pushing for hours </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Forget to turn it back on SSTable count gets bad fast </li></ul></ul></ul></ul><ul><ul><li>Range scanning to locate and remove old data </li></ul></ul><ul><ul><li>Scheduling repairs and compaction </li></ul></ul><ul><ul><li>Mr. Batch enjoys tearing through data </li></ul></ul><ul><ul><ul><li>Writes, tombstones, range scanning, repairs </li></ul></ul></ul><ul><ul><ul><li>Enjoys fast disks for compacting </li></ul></ul></ul>
    25. 25. Questions <ul><ul><li>??? </li></ul></ul>
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×