Cassandra Day Denver 2014: Feelin' the Flow: Analyzing Data with Spark and Cassandra

•

3 likes•3,141 views

Speaker: Rich Beaudoin, Senior Software Engineer at Pearson eCollege In the world of Big Data it's crucial that your data is accessible. Cassandra provides us with a means to reliably store our data, but how can we keep it flowing? That's where Spark steps up to provide a powerful one-two punch with Cassandra to get your data flowing in all the right directions.

Technology

FEELIN' THE
FLOW
GETTING YOUR DATA MOVING
WITH SPARK AND CASSANDRA
Presented by / Rich Beaudoin @RichGBeaudoinOctober 14th, 2014

ABOUT ME...
Sr. Software Engineer at Pearson
Organizer of
Lover of Music
All around solid dude
Distributed Computing Denver

OVERVIEW
What is Spark
The problem it solves
The core concepts
Spark integration with Cassandra
Tables as RDDs
Writing RDDs to Cassandra
Question and Summary

WHAT IS SPARK?
Apache Spark™ is a fast and general engine
for large-scale data processing.
Created by AMPLab at UC Berkeley
Became Apache Top-Level Project in 2014
Supports Scala, Java, and Python APIs

THE PROBLEM, PART
ONE...
Approaches like MapReduce read from, and store to HDFS
...so each cycle of processing incurs latency from HDFS reads

THE PROBLEM, PART
TWO...
Any robust, distributed data processing framework needs fault
tolerance
But existing solutions allow for "fine-grained" (cell level)
updates, which can complicate the handling of faults where
data needs to be rebuilt/recalculated

SPARK ATTEMPTS TO
ADDRESS THESE TWO
PROBLEMS
Solution 1: store intermediate results in memory
Solution 2: introduce a new expressive data abstraction

RDD
A Resilient Distributed Dataset (RDD) is an an
immutable, partioned record that supports
basic operations (e.g. map, filter, join). It
maintains a graph of transformations in order
to enable recovery of a lost partition
*See the RDD white paper for more details

TRANSFORMATIONS
AND ACTIONS
"transformation" creates another RDD, is evaluated lazily
"action" returns a value, evaluated immediately

RDDS ARE EXPRESSIVE
It turns out that coarse-grained operations cover many existing
parrallel computing cases
Consequently, the RDD abstraction can implement existing
systems like MapReduce, Pregel, Dryad, etc.

SPARK CLUSTER
OVERVIEW
Spark can be run with Apache Mesos, HADOOP Yarn, or it's own standalone cluster manager

SPARK AND CASSANDRA
If we can turn Cassandra data into RDDs, and RDDs into
Cassandra data, then the data can start flowing between the
two systems and give us some insight into our data.
allows us to perform the
The Spark Cassandra Connector
transformation from Cassadra table to RDD and then back
again!

FROM CASSANDRA
TABLE TO RDD
import org.apache.spark._
import com.datastax.spark.connector._
val rdd = sc.cassandraTable("music", "albums_by_artist")
Run these commands spark-shell, requires specifying the spark-connector jar on the commandline

SIMPLE MAPREDUCE FOR
RDD COLUMN COUNT
val count = rdd.map(x => (x.get[String]("label"),1)).reduceByKey(_ + _)

SAVE THE RDD TO
CASSANDRA
count.saveToCassandra("music", "label_count",SomeColumns("label", "count"))

CASSANDRA WITH
SPARKSQL
import org.apache.spark.sql.cassandra.CassandraSQLContext
val cc = new CassandraSQLContext(sc)
val rdd = cc.sql("SELECT * from music.label_count")

JOINS!!!
import sqlContext.createSchemaRDD
import org.apache.spark.sql._
case class LabelCount(label: String, count: Int)
case class AlbumArtist(artist: String, album: String, label: String, year: Int)
case class AlbumArtistCount(artist: String, album: String, label: String, year: Int, count
val albumArtists = sc.cassandraTable[AlbumArtist]("music","albums_by_artists").cache
val labelCounts = sc.cassandraTable[LabelCount]("music", "label_count").cache
val albumsByLabelId = albumArtists.keyBy(x => x.label)
val countsByLabelId = labelCounts.keyBy(x => x.label)
val joinedAlbums = albumsByLabelId.join(countsByLabelId).cache
val albumArtistCountObjects = joinedAlbums.map(x => (new AlbumArtistCount(x._2._1.artist,

OTHER THINGS TO
CHECK OUT
Spark Streaming
Spark SQL

THE END
References
Resilient Distributed Datasets: A Fault-Tolerant Abstraction
for In-Memory Cluster Computing
Spark Programming Guide
Apache Spark Website
Datastax Spark Cassandra Connector Documentation

This document discusses using Spark and Cassandra together for distributed data processing. It introduces Spark as a fast engine for large-scale data processing that stores intermediate results in memory. It describes how Spark addresses problems with latency in HDFS reads/writes through RDDs and fault tolerance. It explains how to access Cassandra tables as RDDs, perform transformations on them like joins, and write results back to Cassandra. Examples show mapping Cassandra data to RDDs, running MapReduce jobs, and saving results.

Geek Night - Functional Data Processing using Spark and Scala

Atif Akhtar

Apache Spark is an open-source framework for large-scale data processing. It provides APIs in Java, Scala, Python and R and runs on Hadoop, Mesos, standalone or in the cloud. Spark addresses limitations of Hadoop like lack of iterative algorithms and real-time processing. It provides a more functional API using RDDs that support lazy evaluation, fault tolerance and in-memory computing for faster performance. Spark also supports SQL, streaming, machine learning and graph processing through libraries built on its core engine.

SPARK ARCHITECTURE

GauravBiswas9

Spark is a framework for large-scale data processing. It includes Spark Core which provides functionality like memory management and fault recovery. Spark also includes higher level libraries like SparkSQL for SQL queries, MLlib for machine learning, GraphX for graph processing, and Spark Streaming for real-time data streams. The core abstraction in Spark is the Resilient Distributed Dataset (RDD) which allows parallel operations on distributed data.

Spark

Heena Madan

Spark is an open-source distributed computing framework used for processing large datasets. It allows for in-memory cluster computing, which enhances processing speed. Spark core components include Resilient Distributed Datasets (RDDs) and a directed acyclic graph (DAG) that represents the lineage of transformations and actions on RDDs. Spark Streaming is an extension that allows for processing of live data streams with low latency.

Apache spark - Architecture , Overview & libraries

Walaa Hamdy Assy

This document provides an overview of Apache Spark, an open-source unified analytics engine for large-scale data processing. It discusses Spark's core APIs including RDDs and transformations/actions. It also covers Spark SQL, Spark Streaming, MLlib, and GraphX. Spark provides a fast and general engine for big data processing, with explicit operations for streaming, SQL, machine learning, and graph processing. The document includes installation instructions and examples of using various Spark components.

Spark architecture

GauravBiswas9

The document discusses Apache Spark, an open source cluster computing framework for real-time data processing. It notes that Spark is up to 100 times faster than Hadoop for in-memory processing and 10 times faster on disk. The main feature of Spark is its in-memory cluster computing capability, which increases processing speeds. Spark runs on a driver-executor model and uses resilient distributed datasets and directed acyclic graphs to process data in parallel across a cluster.

Module01

NPN Training

This document provides an overview of Apache Spark, including its history, features, architecture and use cases. Spark started in 2009 at UC Berkeley and was later adopted by the Apache Foundation. It provides faster processing than Hadoop by keeping data in memory. Spark supports batch, streaming and interactive processing on large datasets using its core abstraction called resilient distributed datasets (RDDs).

Map reduce vs spark

Tudor Lapusan

This document compares MapReduce and Spark frameworks. It discusses their histories and basic functionalities. MapReduce uses input, map, shuffle, and reduce stages, while Spark uses RDDs (Resilient Distributed Datasets) and transformations and actions. Spark is easier to program than MapReduce due to its interactive mode, but MapReduce has more supporting tools. Performance benchmarks show Spark is faster than MapReduce for sorting. The hardware and developer costs of Spark are also lower than MapReduce.

Apache Spark is an open source framework for large-scale data processing. It was originally developed at UC Berkeley and provides fast, easy-to-use tools for batch and streaming data. Spark features include SQL queries, machine learning, streaming, and graph processing. It is up to 100 times faster than Hadoop for iterative algorithms and interactive queries due to its in-memory processing capabilities. Spark uses Resilient Distributed Datasets (RDDs) that allow data to be reused across parallel operations.

Introduction to apache spark

Aakashdata

we will see an overview of Spark in Big Data. We will start with an introduction to Apache Spark Programming. Then we will move to know the Spark History. Moreover, we will learn why Spark is needed. Afterward, will cover all fundamental of Spark components. Furthermore, we will learn about Spark’s core abstraction and Spark RDD. For more detailed insights, we will also cover spark features, Spark limitations, and Spark Use cases.

Learning spark ch04 - Working with Key/Value Pairs

phanleson

Beneath RDD in Apache Spark by Jacek Laskowski

Spark Summit

This document provides an overview of SparkContext and Resilient Distributed Datasets (RDDs) in Apache Spark. It discusses how to create RDDs using SparkContext functions like parallelize(), range(), and textFile(). It also covers DataFrames and converting between RDDs and DataFrames. The document discusses partitions and the level of parallelism in Spark, as well as the execution environment involving DAGScheduler, TaskScheduler, and SchedulerBackend. It provides examples of RDD lineage and describes Spark clusters like Spark Standalone and the Spark web UI.

Spark from the Surface

Josi Aranda

Apache Spark is an open-source distributed processing engine that is up to 100 times faster than Hadoop for processing data stored in memory and 10 times faster for data stored on disk. It provides high-level APIs in Java, Scala, Python and SQL and supports batch processing, streaming, and machine learning. Spark runs on Hadoop, Mesos, Kubernetes or standalone and can access diverse data sources using its core abstraction called resilient distributed datasets (RDDs).

Apache spark

Dona Mary Philip

Apache Spark is a lightning-fast cluster computing technology, designed for fast computation. It extends the MapReduce model of Hadoop to efficiently use it for more types of computations, which includes interactive queries and stream processing. Spark is one of Hadoop's subproject developed in 2009 in UC Berkeley's AMPLab by Matei Zaharia. It was Open Sourced in 2010 under a BSD license. It was donated to Apache software foundation in 2013, and now Apache Spark has become a top-level Apache project from Feb-2014. This document shares some basic knowledge about Apache Spark.

Introduction to Apache Spark

Samy Dindane

Apache Spark is a fast distributed data processing engine that runs in memory. It can be used with Java, Scala, Python and R. Spark uses resilient distributed datasets (RDDs) as its main data structure. RDDs are immutable and partitioned collections of elements that allow transformations like map and filter. Spark is 10-100x faster than Hadoop for iterative algorithms and can be used for tasks like ETL, machine learning, and streaming.

Getting Started with Spark Streaming

Alex Apollonsky

Spark Streaming extends Spark to allow processing of real-time data streams. It receives data from sources like Kafka, files, and sockets and divides the streams into micro-batches which are then processed using Spark's RDD transformations and actions. This allows for horizontally scalable, high throughput, and fault-tolerant stream processing. Spark Streaming can also seamlessly integrate with machine learning algorithms in Spark MLlib.

Learning spark ch01 - Introduction to Data Analysis with Spark

phanleson

DUG'20: 02 - Accelerating apache spark with DAOS on Aurora

Andrey Kudryavtsev

This document summarizes accelerating Apache Spark with DAOS (Distributed Asynchronous Object Storage) on Aurora. It describes using DAOS as a Hadoop filesystem for Spark input/output storage and as a shuffle data store. It shows how the DAOS Hadoop filesystem delivers similar throughput to DAOS DFS. It also introduces a DAOS object-based Spark shuffle manager that improves shuffle read throughput by up to 1.5x compared to using the local filesystem, especially for smaller shuffle blocks. Future plans include optimizing the DAOS Spark shuffle manager using async APIs and simplifying user configuration.

Apache Spark 101

Abdullah Çetin ÇAVDAR

This document provides an overview of Apache Spark, including its goal of providing a fast and general engine for large-scale data processing. It discusses Spark's programming model, components like RDDs and DAGs, and how to initialize and deploy Spark on a cluster. Key aspects covered include RDDs as the fundamental data structure in Spark, transformations and actions, and storage levels for caching data in memory or disk.

Apache Spark Introduction and Resilient Distributed Dataset basics and deep dive

Sachin Aggarwal

We will give a detailed introduction to Apache Spark and why and how Spark can change the analytics world. Apache Spark's memory abstraction is RDD (Resilient Distributed DataSet). One of the key reason why Apache Spark is so different is because of the introduction of RDD. You cannot do anything in Apache Spark without knowing about RDDs. We will give a high level introduction to RDD and in the second half we will have a deep dive into RDDs.

Learning Apache Spark by examples

Samuel Yee

Advanced spark training advanced spark internals and tuning reynold xin

caidezhi655

This document provides an overview of Spark's RDD abstraction and the life cycle of a Spark application. It defines RDDs as distributed collections characterized by partitions, dependencies, a compute function, and optional properties like a partitioner. It describes how Spark builds a DAG of stages from transformations and actions, and schedules tasks across executors. The document also covers performance debugging techniques like identifying slow stages, stragglers, garbage collection issues, and profiling tasks locally. Debugging tools discussed include the Spark UI, executor logs, jstack, and YourKit.

Apache Spark overview

DataArt

This document provides an overview of Apache Spark, including how it compares to Hadoop, the Spark ecosystem, Resilient Distributed Datasets (RDDs), transformations and actions on RDDs, the directed acyclic graph (DAG) scheduler, Spark Streaming, and the DataFrames API. Key points covered include Spark's faster performance versus Hadoop through its use of memory instead of disk, the RDD abstraction for distributed collections, common RDD operations, and Spark's capabilities for real-time streaming data processing and SQL queries on structured data.

Spark + Cassandra

Carl Yeksigian

Intro to Apache Spark

Robert Sanders

Spark core

Freeman Zhang

Spark is a distributed data processing framework that uses RDDs (Resilient Distributed Datasets) to represent data distributed across a cluster. RDDs support transformations like map, filter, and actions like reduce to operate on the distributed data in a parallel and fault-tolerant manner. Key concepts include lazy evaluation of transformations, caching of RDDs, and use of broadcast variables and accumulators for sharing data across nodes.

Cassandra Summit 2014: Reading Cassandra SSTables Directly for Offline Data A...

Cassandra Day Denver 2014: Feelin' the Flow: Analyzing Data with Spark and Cassandra

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (14)

Similar to Cassandra Day Denver 2014: Feelin' the Flow: Analyzing Data with Spark and Cassandra

Similar to Cassandra Day Denver 2014: Feelin' the Flow: Analyzing Data with Spark and Cassandra (20)

More from DataStax Academy

More from DataStax Academy (20)

Recently uploaded

Recently uploaded (20)

Cassandra Day Denver 2014: Feelin' the Flow: Analyzing Data with Spark and Cassandra