Cassandra Day SV 2014: Building a Flexible, Real-time Big Data Applications Platform on Apache Cassandra with Kiji
Upcoming SlideShare
Loading in...5
×
 

Cassandra Day SV 2014: Building a Flexible, Real-time Big Data Applications Platform on Apache Cassandra with Kiji

on

  • 493 views

The talk is about the process of adding support for Cassandra in Kiji, our open-source platform for building big-data applications. I start off by describing the Kiji project, how it enables folks to ...

The talk is about the process of adding support for Cassandra in Kiji, our open-source platform for building big-data applications. I start off by describing the Kiji project, how it enables folks to build big-data applications, and (hopefully) get everyone excited about it. Then I talk about the Kiji data model, its origins in HBase (we initially built Kiji on top of HBase), how we updated it to also support Cassandra, what some if the issues were, etc. I get into some detail about our use of the Java driver and its async API, how we translate operations in Kiji into CQL statements, and some enhancements we've made to the Hadoop InputFormat and OutputFormat. I think this talk will be interesting to folks in general, and in particular will be useful for anyone who has an HBase background and is now working with Cassandra.

The Kiji Project is a modular, open-source framework that enables developers to efficiently build real-time Big Data applications. Kiji is built upon popular open-source technologies such as Cassandra, HBase, Hadoop, and Scalding, and contains components that implement functionality critical for Big Data applications, including the following:
Support for evolvable schemas of complex data types
Batch training of machine learning models with Hadoop
Real-time scoring with trained models
Integration with Hive and R
A REST endpoint

Recently, we have updated Kiji to use Cassandra as a backing data store (previously, Kiji worked only with HBase). In this talk, we describe the process of integrating Cassandra and Kiji. Topics we cover include the following:
The Kiji architecture and data model
Implementing the Kiji data model in Cassandra using the Java driver and CQL3
Integrating Cassandra with Hadoop 2.x
Building a flexible middleware platform that supports Cassandra and HBase (including projects that use both simultaneously)
Exposing unique features of Cassandra (e.g., variable consistency) to Kiji users

Statistics

Views

Total Views
493
Views on SlideShare
492
Embed Views
1

Actions

Likes
0
Downloads
10
Comments
0

1 Embed 1

http://www.slideee.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Cassandra Day SV 2014: Building a Flexible, Real-time Big Data Applications Platform on Apache Cassandra with Kiji Cassandra Day SV 2014: Building a Flexible, Real-time Big Data Applications Platform on Apache Cassandra with Kiji Presentation Transcript

    • Building a Flexible, Real-time Big Data Applications Platform on Cassandra with Kiji Cassandra Day Silicon Valley 07 April 2014 Clint Kelly Member of Technical Staff WibiData 1
    • Overview • The Kiji Project • The Kiji data model and KijiSchema • Mapping Kiji to Cassandra • Status and future work • Try it now! 2 Should there be any intro page that talks about WibiData anywhere?
    • The Kiji Project 3
    • 4 ! Want to build this...
    • Have this... 5 ! Want to build this...
    • ! Have this... 6 Want to build this...
    • Open source components • Batch processing – Extract, transform, load – Train machine learning models • Scalable storage – Time-series data • Serialization – Complex data types 7 Hadoop, C*, HBase, Avro KijiSchema KijiMR KijiREST KijiHive KijiScoring KijiExpress
    • KijiSchema • Schemas and data serialization • Complex, atomic data types 8 Hadoop, C*, HBase, Avro KijiSchema KijiMR KijiREST KijiHive KijiScoring KijiExpress record UserLog { long timestamp; int user_id; string url; long session_id; } • Schema evolution • Table metadata
    • Kiji batch components • Scala DSL ➔ describe MapReduce computations • Machine learning library • Hive adapter 9 Hadoop, C*, HBase, Avro KijiSchema KijiMR KijiREST KijiHive KijiScoring KijiExpress
    • Kiji real-time components • REST server • Scoring server 10 Hadoop, C*, HBase, Avro KijiSchema KijiMR KijiREST KijiHive KijiScoring KijiExpress
    • Kiji Summary • Bridge between open-source technologies and real-time, big data applications • Users are building real systems with Kiji now! – Personalized recommendation systems for retail – Energy usage and analytics reporting 11
    • The Kiji data model and KijiSchema 12
    • row 13 Table are composed of rows.
    • entity ID data 14 We call row keys “entity IDs.”
    • data0xfa “bob” 15 We support composite entity IDs (with hashed and unhashed components).
    • info0xfa “bob” songs 16 Data in rows is organized into “column families.”
    • songs: let it be songs: help songs: helter skelter 0xfa “bob” info: email info: payment 17 Column families contain columns, named as “family:qualifier.”
    • songs: let it be songs: help songs: helter skelter 0xfa “bob” info: email info: payment songs: let it besongs: let it besongs: let it be songs: let it be 1396560123 18 Individual columns can have many different timestamped versions.
    • songs: let it be songs: help songs: helter skelter 0xfa “bob” info: email info: payment songs: let it besongs: let it besongs: let it be songs: let it be 1396560123 19 Data values can be complex records record SongPlay { long song_id; int user_rating; long session_id; device_type device; }
    • 20 Locality groups Separate logical organization of data (column families) from physical attributes (caching, compression, etc.) info songs_todayentity ID songs_prev_year
    • 21 Locality groups Separate logical organization of data (column families) from physical attributes (caching, compression, etc.) Need this data ASAP for real-time scoring. Use this data only for batch jobs. info songs_todayentity ID songs_prev_year
    • info songs_todayentity ID songs_prev_year “real_time” (in-memory, uncompressed, TTL = 1 day) “batch” (compressed, TTL = 12mo) 22 Locality groups Always refer to columns by logical name (“family:qualifier”). Need this data ASAP for real-time scoring. Use this data only for batch jobs.
    • KijiSchema summary • Data model similar to Cassandra, HBase, BigTable • Contains time dimension (not present in C*) • Logical and physical organization separate • Complex schemas with Avro 23
    • Mapping Kiji to Cassandra 24
    • Implementation notes 25 • Built for Cassandra 2.0.6+ • Native protocol / Java driver (no Thrift) • Asynchronous API • Assume users have Hadoop, ZooKeeper
    • Mapping a Kiji table ➔ Cassandra • Locality group ➔ Table • Entity ID ➔ Primary key – Hashed components ➔ partition key – Unhashed components ➔ clustering columns • Family, qualifier, timestamp ➔ clustering columns • Cell values ➔ blobs 26 songs: let it be songs: help songs: helter skelter 0xfa “bob” info: email info: payment songs: let it besongs: let it besongs: let it be songs: let it be 1396560123
    • CQL for Kiji locality group CREATE TABLE users_locality_group_fast ( userid bigint, user text, family text, qualifier text, timestamp bigint, value blob, PRIMARY KEY (userid, username, family, qualifier, timestamp) ) WITH CLUSTERING ORDER BY ( username ASC, family ASC, qualifier ASC, timestamp DESC); 27 TODO: Show row diagram, arrows pointing to components?
    • 28 cqlsh:kiji_music>SELECT * FROM kiji_table_users; userid | username | family | qualifier | timestamp | value --------+----------+--------+----------------+-----------+--------------- 0xfa | bob | info | email | 139653249 | 1243970104327 0xfa | bob | songs | abbey road | 139656012 | 0981274331032 0xfa | bob | songs | help | 139625013 | 9074132704129 0xfa | bob | songs | help | 139621359 | 1923079210370 0xfa | bob | songs | help | 139625013 | 4745018223497 0xfa | bob | songs | helter skelter | 139621324 | 7710423974234
    • Physical organization of data on disk 29 songs: let it be songs: help songs: helter skelter 0xfa “bob” info: email info: payment songs: let it besongs: let it besongs: let it be songs: let it be 1396560123 0xfa:bob:info:email:t0:bob@gmail.com 0xfa:bob:info:payment:t1:AMEX1234... 0xfa:bob:songs:let it be:t5:... 0xfa:bob:songs:let it be:t4:… 0xfa:bob:songs:let it be:t2:… 0xfa:bob:songs:help:t2:… 0xfa:bob:songs:helter skelter:t1:… Efficient queries = continuous scans!
    • Kiji queries ➔ CQL queries All data in “info” column family for “bob” ➔ SELECT qualifier, value FROM music WHERE userid=0xfa AND user=‘bob’ AND family=‘info’; 30 songs: let it be songs: help songs: helter skelter 0xfa “bob” info: email info: payment songs: let it besongs: let it besongs: let it be songs: let it be 1396560123
    • Kiji queries ➔ CQL queries Data in “info:email” and last play of “help” for “bob” ➔ SELECT value FROM music WHERE userid=0xfa AND user=‘bob’ AND family=‘info’ AND qualifier=‘email’; SELECT value FROM music WHERE userid=0xfa AND user=‘bob’ AND family=‘songs’ AND qualifier=‘help’ LIMIT 1; 31 songs: let it be songs: help songs: helter skelter 0xfa “bob” info: email info: payment songs: let it besongs: let it besongs: let it be songs: let it be 1396560123
    • Kiji queries ➔ CQL queries All songs played by “bob” on April 2nd ➔ SELECT qualifier, value FROM music WHERE userid=0xfa AND user=‘bob’ AND family=‘songs’ AND timestamp >= 1396396800 AND timestamp <= 1396483200 ALLOW FILTERING; 😱😱 32 songs: let it be songs: help songs: helter skelter 0xfa “bob” info: email info: payment songs: let it besongs: let it besongs: let it be songs: let it be 1396560123
    • Kiji queries ➔ CQL queries 33 songs: let it be songs: help songs: helter skelter 0xfa “bob” info: email info: payment songs: let it besongs: let it besongs: let it be songs: let it be 1396560123 ! Bad Request: PRIMARY KEY part timestamp cannot be restricted (preceding part qualifier is either not restricted or by a non-EQ relation)
    • Queries that do not map well to CQL • Break up into multiple CQL queries – Hooray for Session#executeAsync! • Filter on the client – Potentially very expensive, but functional – Provide warning to user • Educate users about table layout – Layout in previous example is terrible for that query • Most issues related to “time” dimension 34
    • MapReduce • Wrote new InputFormat, OutputFormat • Hadoop 2.x • Multiple C* queries per RecordReader • Does not use Thrift 35
    • Project status and next steps 36
    • Initial release in ~ 2 weeks 37 • Cassandra as part of the Bento Box • Cassandra working in KijiSchema, KijiMR
    • Support in the coming months • Cassandra integration with KijiREST, KijiScoring, KijiExpress, etc. • Expose Cassandra-specific features to users – Variable consistency levels – Load-balancing policies – Diagnostics (e.g., route tracing) • Kiji support in CQLSH – Decode Avro values 38
    • Thanks to Cassandra community • Great help on mailing lists for users, dev, java driver • Webinars, meetups, C* Summit all available online • Free training from DataStax • Very easy to get up-to-speed 39
    • Try it now -- Kiji Bento Box • Latest compatible versions of all components • Hadoop, ZooKeeper, HBase • Cassandra in ~2 weeks 40 www.kiji.org/getstarted Mention hiring?
    • KijiSchema • Schemas and data serialization • Complex data types (e.g., nested maps) • Schema evolution • Metadata • Composite row keys • Transparent paging • Data-definition language, REPL 41 Hadoop, C*, HBase, Avro KijiSchema KijiMR KijiREST KijiHive KijiScoring KijiExpress
    • 42 Schema support Support for complex schemas with Avro record UserLog { long timestamp; int user_id; string url; } KijiSchema allows schema versioning
    • 43 Column name translation •“family:qualifier” -> “A:B” •Saves disk space •Improves performance •User-facing tools translate names •Possible to turn this off
    • Kiji queries ➔ CQL queries All data in family “songs” for user “bob” ➔ SELECT qualifier, value FROM music WHERE userid=0xfa AND user=‘bob’ AND family=‘songs’; 44 songs: let it be songs: help songs: helter skelter 0xfa “bob” info: email info: payment songs: let it besongs: let it besongs: let it be songs: let it be 1396560123