Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
The Economics of SQL on
Hadoop

© 2013 Datameer, Inc. All rights reserved.
Watch the Recording of this Webinar


View the entire recorded webinar at:

http://info.datameer.com/SlideshareEconomics-S...
About our Speakers
John Myers
!
John Myers joined Enterprise Management Associates
in 2011 as senior analyst of the busine...
About our Speakers
Stefan Groschupf!
!
▪  Stefan Groschupf is the co-founder and CEO of

Datameer. He is one of the origin...
About our Speakers
Matt Schumpert!
!
Matt has been working in enterprise software of
over 10 years in various capacities, ...
Agenda
▪  EMA on Current State of the Big Data Industry!
– 
– 
– 
– 
– 

Online Archiving in Practice!
SQL on NoSQL: Metad...
State of Big Data Industry

© 2013 Datameer, Inc. All rights reserved.
Online Archiving is the majority use case for Big
Data projects

Slide 8

© 2013Enterprise Management Associates, Inc.
Moving Beyond select * from tablename
SQL requires a managed set of metadata

Slide 9

© 2013Enterprise Management Associa...
Big Data Platforms have Multiple Uses:
Discovery is a significant portion

Slide 10

© 2013Enterprise Management Associate...
Late Binding Schemas are good for Discovery

Slide 11

© 2013Enterprise Management Associates, Inc.
Free as a Free puppy…

Slide 12

© 2013 Enterprise Management Associates, Inc.
Datameer Demos

© 2013 Datameer, Inc. All rights reserved.
Use Case #1: Semi-Structured Data

▪  Noisy, log-structured data à signal

Slide 14

© 2013 Datameer, Inc. All rights res...
Use Case #1: Semi-Structured Data

▪  Noisy, log-structured data à signal
▪  Extract, cast, & define fields on demand

Slid...
Use Case #1: Semi-Structured Data

▪  Noisy, log-structured data à signal
▪  Extract, cast, & define fields on demand
▪  Pa...
Use Case #1: Semi-Structured Data

▪  Noisy, log-structured data à signal
▪  Extract, cast, & define fields on demand
▪  Pa...
Use Case #1: Semi-Structured Data

▪  Noisy, log-structured data à signal
▪  Extract, cast, & define fields on demand
▪  Pa...
Use Case #2: Text Analytics
▪  Few/no known fields

Slide 19

© 2013 Datameer, Inc. All rights reserved.
Use Case #2: Text Analytics
▪  Few/no known fields
▪  Notion of a record is nebulous / fluid

Slide 20

© 2013 Datameer, Inc...
Use Case #2: Text Analytics
▪  Few/no known fields
▪  Notion of a record is nebulous / fluid
▪  Wrangling and mining

Slide ...
Use Case #2: Text Analytics
▪  Few/no known fields
▪  Notion of a record is nebulous / fluid
▪  Wrangling and mining
▪  “Bag...
Use Case #2: Text Analytics
▪  Few/no known fields
▪  Notion of a record is nebulous / fluid
▪  Wrangling and mining
▪  “Bag...
Use Case #3: Path Analysis 
▪  Key component of clickstream analysis

Slide 24

© 2013 Datameer, Inc. All rights reserved.
Use Case #3: Path Analysis 
▪  Key component of clickstream analysis
▪  Compares each record to the next/previous

Slide 2...
Use Case #3: Path Analysis 
▪  Key component of clickstream analysis
▪  Compares each record to the next/previous
▪  Define...
Use Case #3: Path Analysis 
▪  Key component of clickstream analysis
▪  Compares each record to the next/previous
▪  Define...
Use Case #3: Path Analysis 
▪  Key component of clickstream analysis
▪  Compares each record to the next/previous
▪  Define...
Takeaways

© 2013 Datameer, Inc. All rights reserved.
When NOT to use SQL on Hadoop
▪  Structured Schemas

or “Schema on Write”

Slide 30

© 2013 Datameer, Inc. All rights rese...
When NOT to use SQL on Hadoop
▪  Structured Schemas

or “Schema on Write”
▪  “Realtime” Query
SLAs for operational
or repo...
When NOT to use SQL on Hadoop
▪  Structured Schemas

or “Schema on Write”
▪  “Realtime” Query
SLAs for operational
or repo...
When to use SQL on Hadoop
▪  Unstructured

Datasets and
“Schema on Read”

Slide 33

© 2013 Datameer, Inc. All rights reser...
When to use SQL on Hadoop
▪  Unstructured

Datasets and
“Schema on Read”
▪  Discovery tasks
designed to find new
connection...
When to use SQL on Hadoop
▪  Unstructured

Datasets and
“Schema on Read”
▪  Discovery tasks
designed to find new
connection...
Summary
▪  EMA on Current State of the Big Data Industry
–  Online Archiving in Practice
–  SQL on NoSQL: Metadata
–  Expl...
Call To Action
■  Visit our website
–  www.datameer.com

■  Download our Trial
–  http://www.datameer.com/Datameer-trial.h...
The Economics of SQL on Hadoop
Upcoming SlideShare
Loading in …5
×

The Economics of SQL on Hadoop

889 views

Published on

Watch the recorded event at: http://info.datameer.com/Slideshare-
Economics-SQL-Hadoop.html

As organizations clamor to utilize their new investments in Hadoop ecosystems AND leverage their existing analytical infrastructures, many rush to integrate SQL as a data access layer to leverage existing skill sets and get started faster.

However, this approach relegates Hadoop to a data management and processing platform rather than the storage and compute engine optimized for analytical workloads it was purpose-built to be.

These slides by EMA and Datameer, will discuss the technical limitations of SQL on Hadoop and propose alternative ways to fully maximize Hadoop investments.

You will understanding:

*how SQL negates the inherent benefits of Hadoop
*why technological paradigm changes can sometimes be good
*use cases when SQL on Hadoop makes sense

Published in: Technology
  • Be the first to comment

The Economics of SQL on Hadoop

  1. 1. The Economics of SQL on Hadoop © 2013 Datameer, Inc. All rights reserved.
  2. 2. Watch the Recording of this Webinar View the entire recorded webinar at: http://info.datameer.com/SlideshareEconomics-SQL-Hadoop.html
  3. 3. About our Speakers John Myers ! John Myers joined Enterprise Management Associates in 2011 as senior analyst of the business intelligence (BI) practice area. John has 10+ years of experience working in areas related to business analytics in professional services consulting and product development roles, as well as helping organizations solve their business analytics problems, whether they relate to operational platforms, such as customer care or billing, or applied analytical applications, such as revenue assurance or fraud management. ! Slide 3 © 2013 Datameer, Inc. All rights reserved.
  4. 4. About our Speakers Stefan Groschupf! ! ▪  Stefan Groschupf is the co-founder and CEO of Datameer. He is one of the original contributors to Nutch, the open source predecessor of Hadoop, Stefan has been at the forefront of the Hadoop and Big Data market. Prior to Datameer, Stefan was the co-founder and CEO of Scale Unlimited, which implemented custom Hadoop analytic solutions for HP, Sun, Deutsche Telekom, Nokia and others. Earlier, Stefan was CEO of 101Tec, a supplier of Hadoop and Nutch-based search and text classification software to industry-leading companies such as Apple, DHL and EMI Music. Stefan has also served as CTO at multiple companies, including Sproose, a social search engine company. Slide 4 © 2013 Datameer, Inc. All rights reserved.
  5. 5. About our Speakers Matt Schumpert! ! Matt has been working in enterprise software of over 10 years in various capacities, including sales engineering, strategic alliances and consulting.  ! ! Matt currently runs the pre-sales engineering team at Datameer, supporting all technical aspects of customer engagement through roll-out of customers into production. !  ! Matt holds a BS in Computer Science from the University of Virginia.! Slide 5 © 2013 Datameer, Inc. All rights reserved.
  6. 6. Agenda ▪  EMA on Current State of the Big Data Industry! –  –  –  –  –  Online Archiving in Practice! SQL on NoSQL: Metadata! Exploratory Use Cases! Late Binding Schemas better for Discovery! Economics of Hadoop! ▪  Datameer on how to solve these problems! –  Use Case #1: Semi-Structured Data ! –  Use Case #2: Text Analytics data! –  Use Case #3: Path Analysis! ▪  Takeaways; and Question and Answer! Slide 6 © 2013 Datameer, Inc. All rights reserved.
  7. 7. State of Big Data Industry © 2013 Datameer, Inc. All rights reserved.
  8. 8. Online Archiving is the majority use case for Big Data projects Slide 8 © 2013Enterprise Management Associates, Inc.
  9. 9. Moving Beyond select * from tablename SQL requires a managed set of metadata Slide 9 © 2013Enterprise Management Associates, Inc.
  10. 10. Big Data Platforms have Multiple Uses: Discovery is a significant portion Slide 10 © 2013Enterprise Management Associates, Inc.
  11. 11. Late Binding Schemas are good for Discovery Slide 11 © 2013Enterprise Management Associates, Inc.
  12. 12. Free as a Free puppy… Slide 12 © 2013 Enterprise Management Associates, Inc.
  13. 13. Datameer Demos © 2013 Datameer, Inc. All rights reserved.
  14. 14. Use Case #1: Semi-Structured Data ▪  Noisy, log-structured data à signal Slide 14 © 2013 Datameer, Inc. All rights reserved.
  15. 15. Use Case #1: Semi-Structured Data ▪  Noisy, log-structured data à signal ▪  Extract, cast, & define fields on demand Slide 15 © 2013 Datameer, Inc. All rights reserved.
  16. 16. Use Case #1: Semi-Structured Data ▪  Noisy, log-structured data à signal ▪  Extract, cast, & define fields on demand ▪  Painful/impossible without inspection Slide 16 © 2013 Datameer, Inc. All rights reserved.
  17. 17. Use Case #1: Semi-Structured Data ▪  Noisy, log-structured data à signal ▪  Extract, cast, & define fields on demand ▪  Painful/impossible without inspection ▪  “One-offs” are possible with SQL+UDFs ▪  But better to collaborate with shared “views” Slide 17 © 2013 Datameer, Inc. All rights reserved.
  18. 18. Use Case #1: Semi-Structured Data ▪  Noisy, log-structured data à signal ▪  Extract, cast, & define fields on demand ▪  Painful/impossible without inspection ▪  “One-offs” are possible with SQL+UDFs ▪  But better to collaborate with shared “views” ▪  Examples: ▪  “User-agent” string ▪  URL Parameters ▪  JSON Slide 18 © 2013 Datameer, Inc. All rights reserved.
  19. 19. Use Case #2: Text Analytics ▪  Few/no known fields Slide 19 © 2013 Datameer, Inc. All rights reserved.
  20. 20. Use Case #2: Text Analytics ▪  Few/no known fields ▪  Notion of a record is nebulous / fluid Slide 20 © 2013 Datameer, Inc. All rights reserved.
  21. 21. Use Case #2: Text Analytics ▪  Few/no known fields ▪  Notion of a record is nebulous / fluid ▪  Wrangling and mining Slide 21 © 2013 Datameer, Inc. All rights reserved.
  22. 22. Use Case #2: Text Analytics ▪  Few/no known fields ▪  Notion of a record is nebulous / fluid ▪  Wrangling and mining ▪  “Bag-of-Words” is a sensible start Slide 22 © 2013 Datameer, Inc. All rights reserved.
  23. 23. Use Case #2: Text Analytics ▪  Few/no known fields ▪  Notion of a record is nebulous / fluid ▪  Wrangling and mining ▪  “Bag-of-Words” is a sensible start ▪  Again, frequent inspection is key Slide 23 © 2013 Datameer, Inc. All rights reserved.
  24. 24. Use Case #3: Path Analysis ▪  Key component of clickstream analysis Slide 24 © 2013 Datameer, Inc. All rights reserved.
  25. 25. Use Case #3: Path Analysis ▪  Key component of clickstream analysis ▪  Compares each record to the next/previous Slide 25 © 2013 Datameer, Inc. All rights reserved.
  26. 26. Use Case #3: Path Analysis ▪  Key component of clickstream analysis ▪  Compares each record to the next/previous ▪  Defines/summarizes transitions, not events Slide 26 © 2013 Datameer, Inc. All rights reserved.
  27. 27. Use Case #3: Path Analysis ▪  Key component of clickstream analysis ▪  Compares each record to the next/previous ▪  Defines/summarizes transitions, not events ▪  Supported by list/array types Slide 27 © 2013 Datameer, Inc. All rights reserved.
  28. 28. Use Case #3: Path Analysis ▪  Key component of clickstream analysis ▪  Compares each record to the next/previous ▪  Defines/summarizes transitions, not events ▪  Supported by list/array types ▪  Requires multi-pass queries Slide 28 © 2013 Datameer, Inc. All rights reserved.
  29. 29. Takeaways © 2013 Datameer, Inc. All rights reserved.
  30. 30. When NOT to use SQL on Hadoop ▪  Structured Schemas or “Schema on Write” Slide 30 © 2013 Datameer, Inc. All rights reserved.
  31. 31. When NOT to use SQL on Hadoop ▪  Structured Schemas or “Schema on Write” ▪  “Realtime” Query SLAs for operational or reporting tasks Slide 31 © 2013 Datameer, Inc. All rights reserved.
  32. 32. When NOT to use SQL on Hadoop ▪  Structured Schemas or “Schema on Write” ▪  “Realtime” Query SLAs for operational or reporting tasks ▪  Highly detailed SQL query requirements (SQL-2003) Slide 32 © 2013 Datameer, Inc. All rights reserved.
  33. 33. When to use SQL on Hadoop ▪  Unstructured Datasets and “Schema on Read” Slide 33 © 2013 Datameer, Inc. All rights reserved.
  34. 34. When to use SQL on Hadoop ▪  Unstructured Datasets and “Schema on Read” ▪  Discovery tasks designed to find new connections and new business value Slide 34 © 2013 Datameer, Inc. All rights reserved.
  35. 35. When to use SQL on Hadoop ▪  Unstructured Datasets and “Schema on Read” ▪  Discovery tasks designed to find new connections and new business value ▪  Lower level SQL queries (SQL-99) Slide 35 © 2013 Datameer, Inc. All rights reserved.
  36. 36. Summary ▪  EMA on Current State of the Big Data Industry –  Online Archiving in Practice –  SQL on NoSQL: Metadata –  Exploratory Use Cases –  Late Binding Schemas better for Discovery ▪  Datameer on how to solve these problems –  Use Case #1: Semi-Structured Data –  Use Case #2: Text Analytics –  Use Case #3: Path Analysis Slide 36 © 2013 Datameer, Inc. All rights reserved.
  37. 37. Call To Action ■  Visit our website –  www.datameer.com ■  Download our Trial –  http://www.datameer.com/Datameer-trial.html Slide 37 © 2013 Datameer, Inc. All rights reserved.

×