BigDataCamp 2011

BigDataCamp 2011 Chris K Wensel

Concurrent, Inc.• Founded in Spring of 2008• Cascading core development• Support, Training, & OEM Licensing

So What is Cascading?

In a Nutshell Processing API Integration APIScheduler API Physical Planner Scheduler Alternative Java API to MapReduce with built in Processing Planner and Workload Scheduler

On Many Platforms Processing API Integration API Scheduler API Physical Planner Scheduler Platform• Apache Hadoop • MapR• Amazon Elastic • EMC/GreenPlum MapReduce • and more**

But How is Cascading Used?

RazorFish/BestBuy Java [unit, regression, & integration testing] Processing API Integration API Scheduler API Physical Planner Scheduler Platform• E-Commerce visitor/customer behavior classiﬁcation• Rule processing against proprietary logs• Backend system integration

FlightCaster JVM Language/DSL [scripting, ad-hoc queries, etc] Logical Planner Processing API Integration API Scheduler API Physical Planner Scheduler Platform• They predict ﬂight delays 6 hrs in advance• Created own API/DSL in Clojure• Used to build predictive models

Etsy JVM Language/DSL [scripting, ad-hoc queries, etc] Logical Planner Processing API Integration API Scheduler API Physical Planner Scheduler Platform• Online retailer• Forked own API/DSL in JRuby • Cascading.JRuby - avail on github

What• User behavior on site• Data driven site features • Taste Test • Facebook gift recommender • Suggested Shops • Top Query List • plus many more on the way

BackType JVM Language/DSL [scripting, ad-hoc queries, etc] Logical Planner Processing API Integration API Scheduler API Physical Planner Scheduler Platform• Marketing intelligence• Created Cascalog • an API/DSL in Clojure, avail on github

Ion Flux Java [unit, regression, & integration testing] Processing API Integration APIScheduler API Physical Planner Scheduler Platform Gene sequencing

Who Else?http://concurrentinc.com/casestudies/

How is Cascading Different?

Pig/Hive Query Syntax Extension API Logical Planner Processing API Integration API Scheduler API Physical Planner Scheduler PlatformGreat for ad-hoc queries, but hard to operationalize

Oozie/Azkaban Scheduler Syntax Processing API Integration API Scheduler API Physical Planner Scheduler Platform• Great for gluing command line apps together• JVM scripting language + Cascading is less brittle and with more degrees of freedom

But They are Complementary• No reason Oozie (or Talend) can’t be used to drive Cascading apps• No reason Cascading can’t drive raw MR/ Pig/Hive processes (see Rifﬂe)

Architecture isn’t Innovation collection cleansing processing deliveryevent data signal info knowledge normalization scoring mining The point of computing systems is to make data more valuable Everything else is an implementation detail Copyright Concurrent, Inc. 2011. All rights reserved.

To Do• Support more platforms, including in- memory stream processing• Make Planner more intelligent and leverage more complex data ﬂow topologies• Integrate with more systems and applications Copyright Concurrent, Inc. 2011. All rights reserved.

We are Hiringhttp://www.concurrentinc.com/careers/

BigDataCamp 2011

by cwensel on Jul 20, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

1 Embed 1

Statistics

BigDataCamp 2011 — Presentation Transcript