This document discusses integrating Apache Hadoop and Apache Cassandra. It provides an overview of each technology, describing Hadoop as a framework for distributed processing of large datasets and Cassandra as a distributed database. It then describes a system that was set up with four Cassandra nodes and a Hadoop cluster with Hive and Pig to allow loading sample data into Cassandra using Pig scripts and analyzing the data using MapReduce or Pig. The document notes this open source approach is now available commercially from Datastax Enterprise, which combines Cassandra and Solr into a unified big data platform.

1. • Joyabrata Das
Hadoop and Cassandra Integration
An Open Source Approach

2. • Introduction
• Hadoop + Cassandra Integration
• System Description
• Implementation
• Business alternative
• Thank you
Contents
slide sequence

3. • Apache Hadoop software library is
a framework that allows for the
distributed processing of large data
sets across clusters of computers
using simple programming models
(http://hadoop.apache.org/).
• Apache Cassandra, is a
distributed storage system for
managing very large amounts of
structured data spread out across
many commodity servers, while
providing highly available service
with no single point of failure
(http://planetcassandra.org/).
Hadoop Cassandra
Introduction

4. • Hadoop consists of a Distributed File System (HDFS) at the core
and libraries to support Map Reduce model to write programs
(also by Hive queries or Pig scripts) to do analyze batch oriented
passive data.
• Cassandra is not a conventional database but is more like
Hashtable or HashMap which stores a key/value pair which
allows fast read/write which is crucial for real time data handling.
• While HDFS is a good solution for providing cost-effective storage
for Hadoop implementations devoted to data warehouse systems,
using Cassandra delivers the ability to run analytics on
Cassandra data that comes from line of business applications.
Hadoop + Cassandra Integration
Using Cassandra as backend storage instead of Hadoop Filesystem

5. System Description
• Setup four nodes Cassandra, Hadoop with Hive and Pig cluster in Ubuntu
Linux servers
• All four Cassandra nodes were connected through ring implementing
distributed data replication & no single point of failure to be monitored using
OpsCenter Dashboard.
• Pig partitioner was changed to match cassandra partitioner

6. Implementation
• Generate sample data to be loaded in .csv file
• Create sample keyspace (schema) & columnfamily (table) through
Cassandra Query Language (CQL)
• Write Pig script to format data file into required tuples and start loading
• After successful loading, the same data is read by simple CQL queries as
well as analyzed (using MapReduce or Pig)

7. Business alternative
• This open source approach of
integrating Hadoop with Cassandra
is now commercially available with
Datastax Enterprise Version.
• The Cassandra File System (CFS)
was designed by DataStax
Corporation to easily run analytics
on Cassandra data. Now
implemented as part of DataStax
Enterprise, which combines Apache
Cassandra, and Solr™ together into
a unified big data platform, CFS
provides the storage foundation that
makes running Hadoop-styled
analytics on Cassandra data hassle-
free.

8. Thank You