An introduction to Apache Crunch

Apache Crunch
●

What is it ?

●

How does it work ?

●

Why use it ?

●

Hadoop MapReduce pipelines

●

Scrunch

●

Joins

www.semtech-solutions.co.nz

info@semtech-solutions.co.nz

Apache Crunch – Pipe line
●

Crunch is based on Google's FlumeJava

●

Provides a Java based API for M/R pipelines

●

It uses an MST ( multiple serializable type ) data model

●

Good for processing complex data types

●

Better for “non tuple” data types i.e.
–

Images

–

Audio

–

Seismic data



Apache Crunch – Pipe line
●

What is a Map Reduce Pipe line ?
–

Map

–

Shuffle

–

Reduce

–

Combine

●

Arranged in sequence and / or in parallel

●

Potentially very long chains



Apache Crunch – Scala
●

Scrunch is a Scala wrapper for Apache Crunch

●

Reduced code

●

Functional and OO styles

●

Uses type inferencing for Map / Reduce

●

Incorporates Java Materialize functionality

●

Includes REPL ( read eval print loop )



Apache Crunch – Joins

●

Details of Joins available in Crunch
–

Inner / Outer like SQL joins

–

Same with Left / Right / Full joins

–

MapSide join is an in memory join



Apache Crunch – Performance
●

A light weight API that runs efficiently

●

Crunch is a thin veneer on top of Map Reduce

●

Two implementations available
–
–

●

Hadoop Writeables
Avro

Avro implementation much faster



Apache Crunch – API
●

Data Model

●

Operators

–

Pipeline

–

DoFn

–

MRPipeline

–

CombineFn

–

MemPipeline

–

FilterFn

–

Pcollection

–

Joins

–

Ptable

–

Cartesian

–

PgroupTable

–

Sort

–

Source

–

Secondary Sort

–

Target

–

Pobject

–

Emitter

–

BloomFilters

–

PType



Contact Us
●

Feel free to contact us at
–


–


●

We offer IT project consultancy

●

We are happy to hear about your problems

●

You can just pay for those hours that you need

●

To solve your problems

An introduction to Apache Crunch

Recommended

Recommended

More Related Content

More from Mike Frampton

More from Mike Frampton (20)

Recently uploaded

Recently uploaded (20)

An introduction to Apache Crunch