Your SlideShare is downloading. ×
0
Cassandra & puppet, scaling data at $15 per month
Cassandra & puppet, scaling data at $15 per month
Cassandra & puppet, scaling data at $15 per month
Cassandra & puppet, scaling data at $15 per month
Cassandra & puppet, scaling data at $15 per month
Cassandra & puppet, scaling data at $15 per month
Cassandra & puppet, scaling data at $15 per month
Cassandra & puppet, scaling data at $15 per month
Cassandra & puppet, scaling data at $15 per month
Cassandra & puppet, scaling data at $15 per month
Cassandra & puppet, scaling data at $15 per month
Cassandra & puppet, scaling data at $15 per month
Cassandra & puppet, scaling data at $15 per month
Cassandra & puppet, scaling data at $15 per month
Cassandra & puppet, scaling data at $15 per month
Cassandra & puppet, scaling data at $15 per month
Cassandra & puppet, scaling data at $15 per month
Cassandra & puppet, scaling data at $15 per month
Cassandra & puppet, scaling data at $15 per month
Cassandra & puppet, scaling data at $15 per month
Cassandra & puppet, scaling data at $15 per month
Cassandra & puppet, scaling data at $15 per month
Cassandra & puppet, scaling data at $15 per month
Cassandra & puppet, scaling data at $15 per month
Cassandra & puppet, scaling data at $15 per month
Cassandra & puppet, scaling data at $15 per month
Cassandra & puppet, scaling data at $15 per month
Cassandra & puppet, scaling data at $15 per month
Cassandra & puppet, scaling data at $15 per month
Cassandra & puppet, scaling data at $15 per month
Cassandra & puppet, scaling data at $15 per month
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Cassandra & puppet, scaling data at $15 per month

21,352

Published on

Constant Contact shares lessons learned from DevOps approach to implementing Cassandra to manage social media data for over 400k small business customers. Puppet is the critical in our tool chain. …

Constant Contact shares lessons learned from DevOps approach to implementing Cassandra to manage social media data for over 400k small business customers. Puppet is the critical in our tool chain. Single most important factor was the willingness of Development and Operations to stretch beyond traditional roles and responsibilities.

Published in: Technology
0 Comments
7 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
21,352
On Slideshare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
63
Comments
0
Likes
7
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • “… a highly scalable second-generation distributed database, bringing together Dynamo's fully distributed design and Bigtable'sColumnFamily-based data model.”
  • Operational attributesFault TolerantData is automatically replicated to multiple nodes for fault-tolerance. Replication across multiple data centers is supported. Failed nodes can be replaced with no downtime.DecentralizedEvery node in the cluster is identical. There are no network bottlenecks. There are no single points of failure.ElasticRead and write throughput both increase linearly as new machines are added, with no downtime or interruption to applications.DurableCassandra is suitable for applications that can't afford to lose data, even when an entire data center goes down.Not just key-value. Replication and Consistency are configurable and datacenter aware
  • Can also be configured cross-datacenter.
  • Consistency Level is tunableONEQUORUMALLAt level ONE, one copy makes it to disk synchronously, before caller returns success.Same with reads. One node is read, so can get old data
  • At QUORUM, two of three (QUORUM) written before returningQUORUM Read: Quorum must AGREEFirst two don’t, so wait for the third node to resolve the tie
  • RISKS0.7.x in beta when we started, multiple betas and RCsRDBMS best practices are understood, if they exist for Cassandra how to discover them?Complicated system, lots of knobs, how to tune themMITIGATIONSWe deployed 0.7.2 across 72 servers in 90 minutes two days before we went live. 0.7.3 any day nowMailing lists, read code, file bugsA little about DatastaxStarted with a small app, higher risk ones come later, Mark will talk about monitoring—we have hundreds of graphs
  • Data model: no joins, referential integrity or fixed schema. If you want those things, you have to code them.Rows with millions of columnsThrift: driver-level interface, doesn’t do things an application-level client should do, e.g. failover, retry
  • Contributed bug reports and patches to Cassandra and HectorIncorporated in follow-on releasesNo need to maintain our own fork
  • Mirror modeShort timeoutsLog errors DB2 is still database of record
  • Necessary for success
  • Authentication and Authorization; test basic functionaiity; rapidly deploy new changes
  • Transcript

    • 1. Cassandra & Puppet:
      Scaling data at $15/month
      Constant Contact
      March 2011
      Dave Connors – VP Operations
      Jim Ancona – Systems Architect
      Mark Schena – Manager Systems Automation
    • 2. Constant Contact
      Constant Contact
      2000 – 2010
      Market leader for Small Businesses
      • Email, Event & Survey
      • 3. Over 400k paying customers
      • 4. No. 134 on the Deloitte Technology Fast 500 listing
      Business model
      • Many customers pay as little as $15 a month
      • 5. ~2 million database transactions per minute
    • Constant Contact
      The business problem
    • 6. Constant Contact
      Small Businesses are looking to us for help with Social Media marketing
      • Social Media 10-100 times more data
      • 7. Challenge with our business model
    • The Key Challenge
      The Key Challenge
      Integrate social media data
      • Solution = NoSQL
      • 8. Cost = Low
      • 9. Time to market = ?
    • Implementation
      Implementing NoSQL
      Ops and Dev both face issues
    • Ops
      Dev
    • 15. Apache Cassandra
      Apache Cassandra
      • Developed at Facebook
      • 16. Open sourced in 2008
      • 17. Incubated at Apache
      • 18. Became an Apache top-level project in 2010
      • 19. http://cassandra.apache.org
      • 20. In use at Digg, Facebook, Twitter, Reddit, Rackspace, Cloudkick, Cisco, …
      • 21. Largest production cluster has over 100 TB of data in over 150 machines
    • What is Cassandra?
      What is Cassandra
    • Replication
      Replication
      How many copies of each piece of data
      do we want?
      N=3
    • 28. Consistency LevelONE
      Consistency Level One
      Y
    • 29. Consistency Level Quorum
      X
    • 30. Risks and Mitigation
      Risks and Mitigation
      • Moving target
      • 31. Developer unfamiliarity
      • 32. Operational procedures
      • 33. Reliability concerns
      • 34. Deployment automation
      • 35. Community involvement
      • 36. Training/Consulting
      • 37. Application selection
      • 38. Lots of monitoring
      • 39. Phased rollout
    • Development Challenges
      Development Challenges
      Understanding the data model
      Choosing a client
      Clients available for Java, Python, .NET, Ruby, PHP
      Don’t use Thrift
      Moving target
    • 40. Open Source
      • Not “one neck to wring”
      • 41. Paid support and training is available: http://datastax.com
      • 42. Community
      Mailing lists
      IRC #cassandra at freenode
      • Contribute
    • Phased Rollout
      • Switchable modes
      • 43. Mirroring
      • 44. Dial-able traffic
    • Collaboration
      • Big, complex project
      • 45. Close collaboration
      • 46. Flexible roles
      • 47. Ability to iterate
    • Ops
      Dev
    • 48. “Are you sure you really want that?”
      “Are you sure you really want that?”
      • 3 500G disks
      • 49. 1 250G disk
      • 50. No SWAP
      • 51. RAID Zero Root Partition and Data Storage
      • 52. 32G Memory
    • We will need how many servers?
      We will need how many servers?
    • 53. How many nodes?
      • Quorum = 3
      • 54. Multiple Datacenters = 2
      • 55. Use only half the available disk = 2
      • 56. 12 Servers = ~1 TB Of Data Storage
      • 57. ~6 TB of Data Storage
      72
      x 6 =
      x 2 = 12
      3
      x 2 = 6
    • 58. Ran
      Random Partitioner
    • 59. Tool Chain
      Tool Chain
    • 60. with Puppet
      • Puppet is the shared framework between Operations and Development
      • 61. Versioning of puppet code allows for adoption of development best practices
      • 62. Leverage Domain specific knowledge and skill
      DevOps with Puppet
    • 63. Always Move Forward
      Always Move Forward
    • 64. Operational Efficiencies
      • Remote logging is a requirement
      • 65. Cassandra uses log4j natively
      • 66. Resources not available for remote log4j development
      • 67. Scribed with Puppet provides the solution
      Operational Efficiencies
    • 68. Development takes the Operational Lead
      • Munin
      • 69. JMX trending
      • 70. Identify critical data points
      • 71. Rapid development of graphs
      • 72. Puppet Definitions are used for rapid deployment
    • Sample Munin Graph
    • 73. Puppet Code
      Example: Munin Puppet Code
      define munin::cassandracolumnfamily ( ) {
      include cassandravirtual
      File <| title == "jmxbin" |>
      $confdir="/opt/cassandra-munin-plugins”
      $plugindir="/etc/munin/plugins"
      $target="/opt/cassandra-munin-plugins/jmx_"
      # Match 3 strings separated by periods
      $pattern = '^([^.]*)[.]([^.]*)[.]([^.]*)$'
      $keyspace = regsubst($name, $pattern, '1')
      $columnfamily = regsubst($name, $pattern, '2')
      $file = regsubst($name, $pattern, '3')
      file {"${keyspace}_${columnfamily}_${file}.conf":
      owner => 'root', ensure => 'file', group => 'root', type => 'file',
      path => "${confdir}/${keyspace}_${columnfamily}_${file}.conf",
      mode => '644',
      content => template("munin/attribute_${file}.conf.erb"),
      require => [ Package['munin-node'], File['/opt/cassandra-munin-plugins'], File['jmxquery'], ],
      }
      file {"$plugindir/${keyspace}_${columnfamily}_${file}":
      ensure => 'link', owner => 'root', group => 'root', mode => '511', type => 'link',
      target => "$target",
      require => [ File['/opt/cassandra-munin-plugins'], File["${keyspace}_${columnfamily}_${file}.conf"], File['jmxquery'], Package['munin-node'], ],
    • 74. Conclusion
      Conclusion
      • Cassandra as an appliance
      • 75. Development Best Practices with Life Cycle Management
      • 76. Traditional vs. Today
      • 77. Infrastructure
      4 weeks 4 hours to build 72 nodes
      • Development to Deployment
      9 months 3 months
      • Cost
      Millions 150k
    • 78. Q&A
      Thank You!

    ×