The Next Generation of Big Data Analytics

The Next Generation
of Big Data Analytics

August 22, 2012

© Hortonworks Inc. 2012 © 2012 Teradata Corporation 1

Today’s Speakers

Jim Walker Cesar Rojas Eric Linden
Dir. Product Marketing Dir. Solutions Marketing Technical Marketing
Hortonworks Teradata Aster Teradata Aster


Big Data Changes the Game

Transactions + Interactions
Petabytes
BIG DATA Mobile Web + Observations
Sentiment

User Click Stream
SMS/MMS
= BIG DATA
Speech to Text

Social Interactions & Feeds
Terabytes WEB Web logs
Spatial & GPS Coordinates
A/B testing
Sensors / RFID / Devices
Behavioral Targeting
Gigabytes CRM Business Data Feeds
Dynamic Pricing
Segmentation External Demographics
Search Marketing
Customer Touches User Generated Content
ERP
Megabytes Affiliate Networks
Purchase detail Support Contacts HD Video, Audio, Images
Dynamic Funnels
Purchase record
Offer details Offer history Product/Service Logs
Payment record

Increasing Data Variety and Complexity


Next Generation Data Architecture Drivers

Business •  Enable new business models & drive faster growth (20%+)
Drivers •  Find insights for competitive advantage & optimal returns

•  Data continues to grow exponentially
Technical •  Data is increasingly everywhere and in many formats
Drivers •  Legacy solutions unfit for new requirements growth

Financial •  Cost of data systems, as % of IT spend, continues to grow
Drivers •  Cost advantages of commodity hardware & open source


Fueling Adoption of the Next Generation

Empower the Ecosystem
•  Apache Hadoop has to just work with what
you already have
•  Apache Hadoop must be a seamless part of
holistic data management strategy
•  Leverage existing assets and tools…
Extend them with new and powerful data
Data Platform Services & Open APIs

Hortonworks
Data Platform


Hortonworks Data Platform (HDP)

•  Simplify deployment to get
started quickly and easily

•  Monitor, manage any size cluster
with familiar console and tools

•  Only platform to include data
integration services to interact
1 with any data source

•  Metadata services opens the
platform for integration with
Hortonworks Data Platform existing applications
Delivers enterprise grade functionality on a proven
Apache Hadoop distribution to ease management, •  Dependable high availability
simplify use and ease integration into the enterprise architecture

The only 100% open source data platform for Apache Hadoop


Shift in Paradigm

Classic BI
Structured & Repeatable Analysis

Business determines what IT structures the data to
questions to ask answer those questions
SQL Performance & Structure
“Capture only
what’s needed”
“Capture in case it’s
needed” MapReduce Processing Flexibility

IT delivers platform for Big Data Analytics
storing, refining, & Business explores data for
Multi-structured & Iterative Analysis questions worth answering
analyzing all data sources


Transactions + Interactions + Observations
Audio, Retain runtime models and
Video,
Images
historical data for ongoing 5 Business Web, Mobile, CRM,
refinement & analysis ERP, SCM, …
Transactions
Docs, & Interactions
Text,
XML

Web
Logs,
Clicks
Big Data 4 Data
Social, Refinery Discovery & Classic
Graph, 1 ETL
Feeds Investigative processing
Analytics
Sensors, 3 Share refined
Devices,
RFID
data & runtime 2
Store, aggregate, and models Interactive
transform multi-structured data
Spatial, data to unlock value Business exploration
GPS
Intelligence
& Analytics
Retain historical data to
Events,
Other
unlock additional value 6
Dashboards, Reports,
Visualization, …


Unified Big Data Architecture
•  Engineers
•  Data Scientists
Java, C/C++, Pig, Python, R, SAS,
•  Quants SQL, Excel, BI, Visualization, etc.
•  Business Analysts

Discovery Integrated
Platform Data Warehouse

Capture, Store, Refine

Audio/ Web & Machine
CRM SCM ERP Images Text
Video Social Logs
Sources of data


Next Generation Big Data Analytics
The Data Discovery Cycle

Analytical Idea

Operational DB Operationalize Zero-ETL Data
or EDW or Move On Load/Integration

Evaluate
Results
SQL & non-SQL
Analysis


Key Elements of a Data Discovery Platform

Highly Efficient & Performant Big Data Platform
1 That Allows Quick Iterations

Hybrid Capabilities that Provide both Legacy
2
(SQL, BI) and New (MapReduce) Interfaces

Significant Out-of-the-Box Analytical Apps that
3
Minimize Development

Democratize Big Data & Maximize Enterprise Adoption


Teradata Aster Data Discovery Platform

Analysts Customers Business Users Data Scientists

Your Analytics & Advanced Reporting Applications

Pattern
Matching
Graph Statistical ELT •  50+ pre-built analytic modules
Develop •  Visual IDE; develop apps in hours
Java, C, Python, Perl … •  Many programming languages

SQL SQL-MapReduce •  SQL-MapReduce framework
•  Analyze both non-relational +
Process Platform Services relational data
(e.g. query planning, dynamic
workload management, security …) •  Linear, incremental scalability

•  Commodity-hardware based
Store Relational Relational •  Software only, cloud, or appliance
Row Column •  Relational-data architecture can
be extended for non-relational types

External HDFS Data (Using SQL-H and HCatalog)


Aster MapReduce Portfolio: the App Store of
Big Data
50+ out-of-the-box SQL-MapReduce analytic applications

Path Analysis Text Analysis
Discover patterns in rows of Derive patterns and extract features
sequential data in textual data

Statistical Analysis Segmentation
High-performance processing of Discover natural groupings of data
common statistical calculations points

Marketing Analytics Data Transformation
Analyze customer interactions to Transform data for more advanced
optimize marketing decisions analysis


Aster SQL-H Enables Data Discovery on
Hadoop Data

Aster SQL-H™
A Business User’s Bridge to Analyze Hadoop Data

Aster SQL-H gives analysts and data scientists a better way
to analyze data stored cheaply in Hadoop
•  Allow standard ANSI SQL to Hadoop data

•  Leverage existing BI tool investments

•  Enable 50+ prebuilt SQL-MapReduce Apps and IDE

•  Improve self-sufficiency for analysts going against Hadoop


Analyst Point of View

Gap 1:
Analysts

Engineers Data Scientists Quants Business Analysts

Java, C/C++, Pig, Python, R, SAS, SQL, Excel, BI, Visualization, etc.

MapReduce
(Processing)
Discovery Active Data
Gap 2: File system lacks
Platform Warehouse
optimizers, data locality,
indexes
Database and Analytic Processing Layer

Data Storage and
Refining

Audio/ Web & Machine
Images Text CRM SCM ERP
Video Social Logs


Analyst’s Goal: Get Insights from Data in
Hadoop


Aster MapReduce Portfolio Teradata Analytics Portfolio
Custom Code and
Development

SQL & SQL-MapReduce SQL

MR, Pig, Hive
Teradata Aster Teradata
IT is the optimizer Discovery Platform IDW


Analytics on Hadoop Data with Aster SQL-H


Aster MapReduce Portfolio
Aster MapReduce Portfolio Teradata Analytics Portfolio

SQL-H SQL & MapReduce
SQL & SQL-MapReduce SQL
SQL

Teradata Aster Teradata
Discovery Platform IDW


Aster SQL-H™ Integration with HCatalog
Aster is the execution layer,
all analytical processing is
Aster Layer: SQL-H done with Aster SQL-
MapReduce functions
(no Hive or Hadoop-MR)

Hadoop
Data Filtering

MR

HCatalog is the metadata
Data

Hive HCatalog repository

Pig

HDFS HDFS is the data repository


When to Use What?

•  The best approach by workload and data type
•  Processing as a Function of Schema Requirements by Data Type

Loading and Refining
Analytics
Low Cost Storage Data Pre- Reporting (User-driven,
& Retention Processing, Transformations interactive)
Prep, Cleansing

Stable Teradata / Teradata
Schema Teradata Teradata Teradata
Hadoop (SQL analytics)

Aster Aster
Evolving Aster /
Schema Hadoop (joining with Aster (SQL + MapReduce
Hadoop
structured data) Analytics)

Aster
Format,
No Schema Hadoop Hadoop Hadoop (MapReduce
Analytics)


Customer Churn Prevention

Challenge Cross-Channel
•  Know when churn will occur Customer Interactions
•  Data Mining tools predict probability but do not
17,000 Customers, 1 Month
identify cause events

With Hadoop
•  Capture, retention and transformation of
customer images (e.g. checks) and customer
voice records

With Aster & Teradata
•  SQL-MapReduce listens and predicts the
customer churn event
–  Identifies all interaction patterns prior to
acquisition or attrition

Business Impact
•  10-300x less effort to pinpoint a customer in
the middle of a decision


More Accurate Customer Churn Prevention

Hadoop captures, Aster does path
stores and transform and sentiment
images and call Social &
Web data
analysis with multi-
records structured data

Multi-Structured Raw
Data
Call Data Analysis
Aster Data
Call Center Voice Discovery +
Records Hadoop Image Data
Platform Marketing
Automation
Images & Capture, Retention

Analytic Results
Dimensional Data
Documents
& (Customer
Transformation Retention
Campaign)
Traditional Data Flow Layer
Data Sources

ETL Tools Teradata
Integrated DW


Aster-Hadoop Integration Demo
Churn Attrition


Use Cases: Optimize Outcomes at Scale

Media optimize Content
Intelligence optimize Detection
Investment optimize Algorithms
Advertising optimize Performance
Fraud optimize Prevention
Regulation optimize Compliance
Retail / Wholesale optimize Inventory turns
Manufacturing optimize Supply chains
Healthcare optimize Patient outcomes
Education optimize Learning outcomes
Government optimize Citizen services
Source: Geoffrey Moore. Hadoop Summit 2012 keynote presentation.


Why Hortonworks and Teradata
Familiar business analysis on Apache Hadoop big data
•  50+ advanced SQL-MapReduce functions (Aster MapReduce Portfolio)
•  SQL-MapReduce development environment to build more functions

Straightforward Database to Apache Hadoop Integration
•  ANSI SQL-based interface to standard HCatalog metadata/schema in Hadoop

Interoperability with existing ecosystem & skillsets
•  BI tools (Tableau, MicroStrategy, Cognos), ETL tools, SQL analysts & existing
applications

Ease of maintenance, skillset and tools compliant
•  Leverage existing DBA skill-sets without additional overhead
•  Apache Ambari provides management and monitoring of the Hadoop cluster and
integrates with current administration tools


Learn More

Big Analytics Best Practices Jim Walker
Hortonworks
www.asterdata.com/BigAnalyticsSeries jim@hortonworks.com

Cesar Rojas
Apache Hadoop Teradata Aster
cesar.rojas@teradata.com
& the Big Data Refinery
www.hortonworks.com Eric Linden
Teradata Aster
eric.linden@teradata.com

Twitter: @hortonworks @asterdata @jaymce


The Next Generation of Big Data Analytics

More Related Content

What's hot

Similar to The Next Generation of Big Data Analytics

More from Hortonworks

Recently uploaded

The Next Generation of Big Data Analytics