Nyc summit intro_to_cassandra
Upcoming SlideShare
Loading in...5
×
 

Nyc summit intro_to_cassandra

on

  • 2,307 views

Introduction to Apache Cassandra from Cassandra NYC summit

Introduction to Apache Cassandra from Cassandra NYC summit

Statistics

Views

Total Views
2,307
Views on SlideShare
2,307
Embed Views
0

Actions

Likes
1
Downloads
94
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as OpenOffice

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • TODO: need fb logo
  • TODO: need fb logo

Nyc summit intro_to_cassandra Nyc summit intro_to_cassandra Presentation Transcript

    • Java, Big Data and Apache Cassandra
      Nate McCall [email_address] @zznate
    • Apache Cassandra: Origins in big data
    • Apache Cassandra: Origins in big data
    View slide
  • But first... the CAP Theorem C onsistency A vailability P artition Tolerance “ Thou shalt have but 2” - Conjecture made by Eric Brewer in 2000 - Published as formal proof in 2002 - See: http://en.wikipedia.org/wiki/CAP_theorem for more View slide
  • CAP Theorem: Cassandra Style - Explicit choice of partition tolerance and availability. - Opt for more consistency at the cost of availability Consistency is tunable (per operation)
    • Apache Cassandra Concepts
    - No read before write - Merge on read - Idempotent - Schema Optional - All nodes share the same roll - Still performs well with larger-than-memory data sets
  • Generally complements another system(s) (Not intended to be one-size-fits-all) *** You should always use the right tool for the right job anyway
  • How does this differ from an RDBMS?
  • How does this differ from an RDBMS? Substantially.
  • vs. RDBMS - No Joins Unless: - you do them on the client - you do them via Map/Reduce
  • vs. RDBMS - Schema Optional (Though you can add meta information for validation and type checking) *** Supports secondary indexes too: “ … WHERE state = 'TX' ”
  • vs. RDBMS - Prematerialized and Transaction-less - No ACID transactions - Limited support for ad-hoc queries
  • vs. RDBMS - Prematerialized and Transaction-less - No ACID transactions - Limited support for ad-hoc queries *** You are going to give up both of these anyway when you shard an RDBMS ***
    • vs. RDBMS - Facilitates Consolidation
    It can be your caching layer * Off-heap cache (provided you install JNA) It can be your analytics infrastructure * true map/reduce * pig driver * hive driver coming soon
  • vs. RDBMS - Shared-Nothing Architecture Every node plays the same role: no masters, no slaves, no special nodes *** No single point of failure
    • vs. RDBMS - Real Linear Scalability
    Want 2x performance? Add 2x nodes (with no downtime!)
    • vs. RDBMS - Performance
    Reads on par with writes
    • Clustering
    • Clustering
    Consistent Hashing FTW: - No fancy shard logic or tedious management of such required - Ring ownership continuously “gossiped” between nodes - Any node can act as a “coordinator” to service client requests for any key * requests forwarded to the appropriate nodes by coordinator transparently to the client
    • Clustering
    Single node cluster (easy development setup) - one node owns the whole hash range
    • Clustering
    Two node cluster - Key range divided between nodes
    • Clustering
    Consistent Hashing: md5(“zznate”) = “C”
    • Clustering: The Client's Perspective
    Client Read: get(“zznate”) md5 = “C”
    • Clustering – Scale Out
    • Clustering – Scale Out
    • Clustering – Scale Out
    • Clustering - Multi-DC
    • Clustering - Reliability
    • Clustering - Reliability
    • Clustering - Reliability
    • Clustering - Reliability
    • Clustering - Multi-Datacenter
    • Clustering – Multi-DC Reliability
    • Storage (Briefly)
    • Storage (Briefly)
      Understanding the on-disk format is extremely helpful in designing your data model correctly
    • Storage - SSTable
      - SSTables are immutable (“Merge on read”)
    • - Newest timestamp wins
    • Storage – Compaction
      Merge SSTables – keeping count down making Merge on Read more efficient
    • Discards Tombstones (more on this later!)
    • Data Model
    • Data Model
      "...sparse, persistent, distributed, multi-dimensional sorted map."
    • (The “Bigtable” paper)
    • Data Model
      Keyspace
    • - Collection of Column Families
    • - Controls replication
    • Column Family
    • - Similar to a table
    • - Columns ordered by name
    • Data Model – Column Family
      Static Column Family
    • - Model my object data
    • Dynamic Column Family
    • - Pre-calculated query results
    • Nothing stopping you from mixing them!
    • Data Model – Static CF
      GOOG
      AAPL
      NFLX
      NOK
    • price: 589.55
    • price: 401.76
      price: 78.73
      name : Google
      name : Apple
      name : Netflix
      price: 6.90
      name : Nokia
      exchange : NYSE
      Stocks
    • Data Model – Prematerialized Query
      StockHist
      10/25/2011: 6.71
      GOOG
      AAPL
      NFLX
      NOK
      10/24/2011: 6.76
      10/21/2011: 6.61
      10/25/2011: 77.37
      10/24/2011: 118.84
      10/21/2011: 117.04
      10/25/2011: 397.77
      10/24/2011: 405.77
      10/21/2011: 392.87
      10/25/2011: 583.16
      10/24/2011: 596.42
      10/21/2011: 590.49
  • Data Model – Prematerialized Query Additional examples: Timeline of tweets by a user Timeline of tweets by all of the people a user is following List of comments sorted by score List of friends grouped by state
    • API Operations
  • Five general categories
      Retrieving Writing/Updating/Removing (all the same op!)
        Increment counters
      Meta Information Schema Manipulation CQL Execution
  • Big Data Fun and Hijinks
      - Hadoop integration - Pig Integration - Hive Integration * open source version coming soon * available in DataStax Enterprise
  • Big Data: Map/Reduce Integration Cassandra Implementations of: - InputFormat and OutputFormat - RecordReader and RecordWriter - InputSplit for Column Families *** See org.apache.cassandra.hadoop package and examples for more
  • Big Data: Pig Integration grunt> name_group = GROUP score_data BY name PARALLEL 3; grunt> name_total = FOREACH name_group GENERATE group, COUNT(score_data.name), LongSum(score_data.score) AS total_score; grunt> ordered_scores = ORDER name_total BY total_score DESC PARALLEL 3; grunt> DUMP ordered_scores;
  • Using a Client Hector Client: http://hector-client.org - Most popular Java client - In use at very large installations - A number of tools and utilities built on top - Very active community - MIT Licensed *** like any open source project fully dependent on another open source project it has it's worts
    • Sample Project for Experimenting
    https://github.com/zznate/cassandra-tutorial https://github.com/zznate/hector-examples Built using Hector Really basic – designed to be beginner level w/ very few moving parts Modify/abuse/alter as needed *** Descriptions of what is going on and how to run each example are in the Javadoc comments. 
    • Hector: ColumnFamilyTemplate
    Familiar, type-safe approach - based on template-method design pattern - generic: ColumnFamilyTemplate<K,N> (K is the key type, N the column name type) ColumnFamilyTemplate template = new ThriftColumnFamilyTemplate(keyspaceName, columnFamilyName, StringSerializer.get(), StringSerializer.get()); *** (no generics for clarity)
    • Hector: ColumnFamilyTemplate
    new ThriftColumnFamilyTemplate(keyspaceName, columnFamilyName, StringSerializer.get(), StringSerializer.get()); Key Format Column Name Format - Cassandra calls this a “comparator” - Remember: defines column order in on-disk format
  • Hector: ColumnFamilyTemplate ColumnFamilyResult<String, String> res = cft.queryColumns(&quot;zznate&quot;); String value = res.getString(&quot;email&quot;); Date startDate = res.getDate(“startDate”); Key Format Column Name Format
  • Hector: ColumnFamilyTemplate ColumnFamilyResult wrapper = template.queryColumns(&quot;zznate&quot;, &quot;patricioe&quot;, &quot;thobbs&quot;) ; while (wrapper.hasNext() ) { emails.put(wrapper.getKey(), wrapper.getString(&quot;email&quot;)); ... Querying multiple rows
  • Hector: ColumnFamilyTemplate ColumnFamilyResult wrapper = template.queryColumns(&quot;zznate&quot;, &quot;patricioe&quot;, &quot;thobbs&quot;); while ( wrapper.hasNext() ) { emails.put(wrapper.getKey(), wrapper.getString(&quot;email&quot;)); ... Iterating over results
  • Hector: ColumnFamilyTemplate ColumnFamilyUpdater updater = template.createUpdater(&quot;zznate&quot;); updater.setString(&quot;companyName&quot;,&quot;DataStax&quot;); updater.addKey(&quot;sergek&quot;); updater.setString(&quot;companyName&quot;,&quot;PrestoSports&quot;); template.update(updater); Insert: Creating an updater for a key
  • Hector: ColumnFamilyTemplate ColumnFamilyUpdater updater = template.createUpdater(&quot;zznate&quot;); updater.setString(&quot;companyName&quot;,&quot;DataStax&quot;); updater.addKey(&quot;sergek&quot;); updater.setString(&quot;companyName&quot;,&quot;PrestoSports&quot;); template.update(updater); Insert: Adding Multiple Rows
  • Hector: ColumnFamilyTemplate ColumnFamilyUpdater updater = template.createUpdater(&quot;zznate&quot;); updater.setString(&quot;companyName&quot;,&quot;DataStax&quot;); updater.addKey(&quot;sergek&quot;); updater.setString(&quot;companyName&quot;,&quot;PrestoSports&quot;); template.update(updater); Insert: Invoking Batch Execution
  • Hector: ColumnFamilyTemplate template.deleteColumn(&quot;zznate&quot;, &quot;notNeededStuff&quot;); template.deleteColumn(&quot;zznate&quot;, &quot;somethingElse&quot;); template.deleteColumn(&quot;patricioe&quot;, &quot;aDifferentColumnName&quot;); ... template.deleteRow(“someuser”); template.executeBatch(); Deleting Data: Single Column
  • Hector: ColumnFamilyTemplate template.deleteColumn(&quot;zznate&quot;, &quot;notNeededStuff&quot;); template.deleteColumn(&quot;zznate&quot;, &quot;somethingElse&quot;); template.deleteColumn(&quot;patricioe&quot;, &quot;aDifferentColumnName&quot;); ... template.deleteRow(“someuser”); template.executeBatch(); Deleting Data: Whole Row
    • Deletion
    • Deletion
    • Again: Every mutation is an insert!
    • - Merge on read
    • - Sstables are immutable
    • - Highest timestamp wins
    • Deletion – As Seen by CLI
      [default@Tutorial] list Portfolio;
    • Using default limit of 100
    • -------------------
    • RowKey: 12783
    • => (column=GOOG, value=30, timestamp=1310340410528000)
    • -------------------
    • RowKey: 15736
    • => (column=AAPL, value=20, timestamp=1310143852392000)
    • => (column=NOK, value=90, timestamp=1310143852444000)
    • => (column=IBM, value=50, timestamp=1310143852448000)
    • => (column=GOOG, value=5, timestamp=1310143852453000)
    • => (column=INTC, value=200, timestamp=1310143852457000)
    • Deletion – As Seen by CLI
      [default@Tutorial] list Portfolio;
    • Using default limit of 100
    • -------------------
    • RowKey: 12783
    • -------------------
    • RowKey: 15736
    • => (column=AAPL, value=20, timestamp=1310143852392000)
    • => (column=NOK, value=90, timestamp=1310143852444000)
    • => (column=IBM, value=50, timestamp=1310143852448000)
    • => (column=GOOG, value=5, timestamp=1310143852453000)
    • => (column=INTC, value=200, timestamp=1310143852457000)
    • Deletion – FYI
      mutator.addDeletion(&quot;14100&quot;, &quot;INTC&quot;, 75, stringSerializer);
      Does not exist? You just inserted a tombstone!
      Sending a deletion for a non-existing row:
      [default@Tutorial] list Portfolio;
    • Using default limit of 100
    • . . .
    • -------------------
    • RowKey: 14100
    • -------------------
    • . . .
    • Integrating with existing patterns
    • Integrating with existing patterns
    • “ Yes.”
    • Integrating with existing patterns
    • <bean id=&quot;cassandraHostConfigurator&quot;
    • class=&quot;me.prettyprint.cassandra.service.CassandraHostConfigurator&quot;>
    • <constructor-arg value=&quot;localhost:9170&quot;/>
    • </bean>
    • <bean id=&quot;cluster&quot; class=&quot;me.prettyprint.cassandra.service.ThriftCluster&quot;>
    • <constructor-arg value=&quot;TestCluster&quot;/>
    • <constructor-arg ref=&quot;cassandraHostConfigurator&quot;/>
    • </bean>
    • <bean id=&quot;consistencyLevelPolicy&quot; class=&quot;me.prettyprint.cassandra.model.ConfigurableConsistencyLevel&quot;>
    • <property name=&quot;defaultReadConsistencyLevel&quot; value=&quot;ONE&quot;/>
    • </bean>
    • <bean id=&quot;keyspaceOperator&quot; class=&quot;me.prettyprint.hector.api.factory.HFactory&quot;
    • factory-method=&quot;createKeyspace&quot;>
    • <constructor-arg value=&quot;Keyspace1&quot;/>
    • <constructor-arg ref=&quot;cluster&quot;/>
    • <constructor-arg ref=&quot;consistencyLevelPolicy&quot;/>
    • </bean>
    • <bean id=&quot;simpleCassandraDao&quot; class=&quot;me.prettyprint.cassandra.dao.SimpleCassandraDao&quot;>
    • <property name=&quot;keyspace&quot; ref=&quot;keyspaceOperator&quot;/>
    • <property name=&quot;columnFamilyName&quot; value=&quot;Standard1&quot;/>
    • </bean>
    • Integrating with existing patterns
    • Hector Object Mapper (simple, JPA 1.0-style annotations):
    • https://github.com/rantav/hector/wiki/Hector-Object-Mapper-%28HOM%29
    • Hector JPA (experimental open-jpa implementation):
    • https://github.com/riptano/hector-jpa
    • Integrating with existing patterns
    • private static final String STOCK_CQL =
    • “ select price FROM Stocks WHERE KEY = ?&quot;;
    • jdbcTemplate.query(STOCK_CQL, stockTicker,
    • new RowMapper<Stock>() {
    • public Stock mapRow(ResultSet rs, int row) throws SQLException {
    • CassandraResultSet crs = (CassandraResultSet)rs;
    • Stock stock = new Stock();
    • stock.setTicker(new String(crs.getKey()));
    • stock.setPrice(crs.getDouble(&quot;price&quot;));
    • return stock;
    • }
    • });
    • Integrating with existing patterns
    • private static String UPDATE_PORTOFOLIO_CQL =
    • &quot; update Portfolios set ? = ? where KEY = ? &quot;;
    • jdbcTemplate.update(UPDATE_PORTFOLIO_CQL,
    • new Object[] {position.getTicker(),
    • position.getCount(),
    • portfolio.getName()});
    • Integrating with existing patterns
    • private static final String UPDATE_PORT_CQL =
    • &quot;update Portfolios set ? = ? where KEY = ?&quot;;
    • jdbcTemplate.batchUpdate(UPDATE_PORT_CQL,
    • new BatchPreparedStatementSetter() {
    • public void setValues(PreparedStatement ps, int index) throws SQLException {
    • Position pos = portfolio.getConstituents().get(index);
    • ps.setString(1, pos.getTicker());
    • ps.setLong(2, pos.getShares());
    • ps.setString(3,portfolio.getName());
    • }
    • public int getBatchSize() {
    • return portfolio.getConstituents().size();
    • }
    • });
    • Putting it Together
    • Take control of consistency
    • If you do need a high degree of consistency, use thresholds to trigger different behavior
    • - Bank account:
    • “ on values over $10,000, wait to here from all replicas”
    • - Distributed Shopping Cart:
    • Show a confirmation page to verify order resolution
    • *** What is your appetite for risk?
  • Uniquely identify operations in the application
    • Facilitates idempotent behavior and out-of-order execution
    • Denormalization
    • The point of normalization is to avoid update anomalies
    • ***But In an append-only system, we don't do updates
    • Summary
    • - Take advantage of strengths
    • - Look for idempotence and asynchronicity in your business processes
    • - If it's not in the API, you are probably doing it wrong
    • - Seek death is still possible if you model incorrectly
    • Questions
      Nate McCall [email_address] @zznate
    • Development Resources
    Hector Documentation http://hector-client.org
    • Cassandra Maven Plugin http://mojo.codehaus.org/cassandra-maven-plugin/
    • CCM localhost cassandra cluster https://github.com/pcmanus/ccm
    • OpsCenter http://www.datastax.com/products/opscenter
      Cassandra AMIs https://github.com/riptano/CassandraClusterAMI
    • Additional Resources
    • DataStax Documentation: http://www.datastax.com/docs/0.8/index
    • Apache Cassandra project wiki: http://wiki.apache.org/cassandra/
    • “ The Dynamo Paper”
    • http://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf
    • P. Helland. Building on Quicksand
    • http://arxiv.org/pdf/0909.1788
    • P. Helland. Life Beyond Distributed Transactions
    • http://www.ics.uci.edu/~cs223/papers/cidr07p15.pdf
    • S. Anand. “Netflix's Transition to High-Availability Storage Systems”
    • http://media.amazonwebservices.com/Netflix_Transition_to_a_Key_v3.pdf
    • “ The Megastore Paper”
    • http://research.google.com/pubs/archive/36971.pdf