The Comprehensive Approach: A Unified Information Architecture

Eric.kavanagh@bloorgroup.com

Twitter Tag: #briefr

!   Reveal the essential characteristics of enterprise
software, good and bad

!   Provide a forum for detailed analysis of today s
innovative technologies

!   Give vendors a chance to explain their product to
savvy analysts

!   Allow audience members to pose serious questions...
and get answers!


!   May: Analytics
!   June: Intelligence
!   July: Disruption
!   August: Analytics
!   September: Integration
!   October: Database


!   Analytics is, and always has been, about discovering insights
that lead to better business decisions. The range of
technologies and use cases that inhabit this area is wide:
statistical analysis, data and process mining, predictive
analytics and modeling, and complex event processing.

!   What is now referred to as Big Data has pushed analytics
beyond the capabilities of traditional solutions. “Big
Analytics” has organizations diving into large heaps of data
that previously was not available or usable.

!   The growing volume, variety, velocity and complexity of
data has proven to be a major challenge to organizations
who leverage analytics to maintain a competitive edge.


Dr. Richard Hackathorn is a well-known
industry analyst, technology innovator
and international educator. He has
pioneered innovations in database
management, decision support and data
warehousing. Richard has published
numerous articles, presented at leading
industry conferences, and conducted
professional seminars in eighteen
countries. He has written three books,
entitled Enterprise Database
Connectivity, Using the Data Warehouse
(with William H. Inmon), and Web
Farming for the Data Warehouse.
Richard taught at the Wharton School
and at the University of Colorado.


!   Teradata is known for its analytic data solutions with
a focus on integrated data warehousing, big data
analytics and business applications.

!   It offers a broad suite of technology platforms and
solutions, and a wide range of data management
applications and data mining capabilities.

!   Teradata features Teradata Aster is its MapReduce
platform to handle big data and big analytics on
multi-structured data.


Chris Twogood is Vice President of
Product and Services Marketing for
Teradata Corporation. He is
responsible for marketing products
(database, utilities, and platform),
and services (professional and
customer services), plus technical
field sales support. Chris has twenty-
five years of experience in the
computer industry specializing in
Data Warehousing, Decision Support,
Customer Management and Appliance
platforms. Chris has held roles that
span Strategy, Application Definition,
Marketing, Product Requirements/
Management, Platform Solutions and
Product Marketing.


Big Data: From Transactions to Interactions

BIG DATA
User Generated
Social Network
Content
Mobile Web
External
User Click Stream Sentiment
Demographics

Web logs
WEB A/B testing Business Data Feeds

Offer history Dynamic Pricing
HD Video
Affiliate Networks
CRM Speech to Text
Segmentation
Search marketing
Offer details
Product/Service Logs
ERP Behavioral Targeting
Customer Touches
Purchase detail
Purchase record Support Contacts Dynamic Funnels SMS/MMS
Payment record

Increasing data variety and complexity

10 Confidential and proprietary. Copyright © 2012 Teradata Corporation.

Unified Big Data Architecture
Bridging Classic & Big Data Worlds

Classic BI
Structured & Repeatable Analysis

Business determines what IT structures the data to
questions to ask answer those questions
SQL performance and structure
“Capture only what’s
needed”

MapReduce processing flexibility

IT delivers a platform for Big Data Analytics
storing, refining, and Business explores data for
Multi-structured & Iterative Analysis
analyzing all data sources questions worth answering

“Capture in case it’s
needed”

Need for a Unified Big Data Architecture for New Insights
Enabling All Users for Any Data Type from Data Capture to Analysis

Java, C/C++, Pig, Python, R, SAS, SQL, Excel, BI, Visualization, etc.

Reporting and Execution
Discover and Explore
in the Enterprise

Capture, Store and Refine

Audio/ Web & Machine
Images Docs Text CRM SCM ERP
Video Social Logs


Unified Big Data Architecture for the Enterprise

Engineers Data Scientists Quants Business Analysts

Java, C/C++, Pig, Python, R, SAS, SQL, Excel, BI, Visualization, etc.

ANALYTICS

Discovery Platform Active Data Warehouse

Capture, Store, Refine

Audio/ Web & Machine
Images Text CRM SCM ERP
Video Social Logs


Analyst’s Goal: Get Insights from Data in Hadoop


Aster MapReduce Portfolio Teradata Analytics Portfolio
Custom Code and
Development

SQL & MapReduce SQL

MR, Pig, Hive
Teradata Aster Teradata
IT is the optimizer MapReduce Platform IDW

HDFS


Analytics on Hadoop Data


Aster MapReduce Portfolio
Aster MapReduce Portfolio Teradata Analytics Portfolio

SQL & MapReduce& MapReduce
SQL SQL
SQL

Teradata Aster Teradata
MapReduce Platform IDW

HDFS


What’s Technically Different in Big Data Analytics
Variety of data types requires different schemas
•  Data that uses a stable schema (structured)
-  Data from packaged business processes with well-defined & known attributes
(e.g., ERP data, Inventory Records, Supply Chain records, …)

•  Data that has an evolving schema (semi-structured)
-  Data generated by machine processes; known but changing set of attributes
(e.g., Web logs, CDRs, Sensor logs, JSON, Social profiles, Twitter feeds, …)

•  Data that has a format, but no schema (unstructured)
-  Data captured by machines with well-defined format, but no semantics
(e.g., images, videos, web pages, PDF documents, …)
-  Semantics can be extracted from raw data by interpreting the format and
extracting semantics
(e.g., shapes from video, face recognition in images, logo detection, …)
-  Sometimes format data is accompanied by meta-data that can have (Stable
Schema or Evolving Schema) – that needs to be classified and treated
separately


When to Use Which?
The best approach by workload and data type
Processing as a Function of Schema Requirements by Data Type

Loading and Refining
Low Cost Analytics
Storage & Data Pre-Processing, Reporting (User-driven,
Retention Prep, Cleansing Transformations interactive)

Stable Teradata / Teradata
Teradata Teradata Teradata
Schema Hadoop (SQL analytics)

Aster Aster
Evolving Aster /
Hadoop (joining with Aster (SQL + MapReduce
Schema Hadoop
structured data) Analytics)

Aster
Format,
Hadoop Hadoop Hadoop (MapReduce
No Schema
Analytics)


Architecture Flexibility – Stable Schema

Solid State
Drives

300-600
GB drives

High Capacity
Drives

Extreme Data Data Warehouse Active Enterprise Data
Appliance Appliance Warehouse

Low Cost Storage
& Retention
Load, Data Prep &
Refining

Transformation

Low Latency, Minimize Data
High volume data CPU Intense
Movement/Complexity,
Benefits storage, light transformations, medium
transformation aligned to
transformations volume data storage
reference data

Automatic compression
Compression Software compression Compress on cold
engines

18 List Price/TB
5/29/12 $4K $11K
Teradata Copyright ©2012 $30K*
* price/TB on cold storage only

Unified Big Data Architecture and Data Flow
Enabling a Data-Driven Business

Transaction Architecture

Traditional
Data Sources
Business
ETL
SQL Analytics Applications

Dimensional Data

Analytic Results
Interaction Architecture

Multi-Structured Analytic
Raw Data Tools
& Users
Sensors, Scientific Unified Analytic Access
and Geospatial Data

Iterative
Store & Discovery
Social Media Refine & Analytics

Unified Big Data Architecture


Thinking Beyond the
Enterprise Data Warehouse

Richard Hackathorn
richardh@bolder.com

© Bolder Technology, Inc. 2012

A New Ballgame!

•  Big Data is forcing us to rethink the goals
and architecture for data warehousing

•  Traditional EDW is no longer sufficient
§  Exclusive collection of corporate information
§  Striving toward a single version of truth
§  Only structured data has business value
§  Predefined questions are the norm

•  We are now facing a new set of issues!

© Bolder Technology, Inc. 2012 Slide 23

Issue: Exclusive to Inclusive

•  All data can not be managed within the
boundaries of the EDW
§  Too much and too fast
§  Too complex and changing
§  Controlled by others
§  New data sources are critical
§  Short-lived data sources are also critical
•  Need to be more agile, flexible, responsive
•  Requiring ‘smart’ curating of new sources
§  What should be captured, stored, and retained?
•  Requiring ‘smart’ data exploration


Issue: Ever-Changing Multiple Truths

•  “More things in heaven and earth than
are dreamt of in your philosophy”
§  IOW we do not know what we do not know!

•  Example: multiple personalities for the
same customer

•  Business semantic analysis is critical and
continuous activity


Issue: Discovering Structure

•  Need for a constant refining of all data
§  Constantly maturing data by enhancing,
compressing, and structuring

•  Business value comes from leveraging
structured data into process variations
§  What do you do differently with what you know?
§  Analytics and data mining add structure


•  An interesting (and seldom discussed) facet of Big Data is the
emerging applications that are NOT social networking analytics on
web logs and website behaviors. What are the ‘killer’ apps in this
area? Do they involve the “Internet of Things”?

•  Big Data is big in volume and in variety. It is also big in velocity.
There is a lot per second…per minute…per day. How should a
unifying architecture handle the velocity of Big Data?

•  Many are trying to “Capture in case it is needed” as their approach
to Big Data. But, can you capture all the data? At what point does
cost of data capture/storage exceed the business benefits? How do
you decide what to capture, store, and retain?

•  Data exploration is an increasingly popular term. How does it differ
from data analysis? Can you really find useful information through
data exploration when you do not know what you are looking for?
Examples?


•  When you unify the architecture for Big Data (as contrasted with
isolated islands of Big Data applications), the data needs to move
through several physical stores. Given the volume and velocity of
data flows, can/should Big Data be duplicated in multiple stores?

•  What is the difference between the Hadoop (Hive, etc) system and
the Teradata Aster system? Could you use both for analytics? Do you
need both in your unifying architecture?

•  Are the ‘traditional’ BI tools (like BusinessObjects, Cognos) relevant
to Big Data analytics? Are they needed in companies that are heavily
Big Data? Are they evolving and expanding to incorporate the new
approaches and techniques required for Big Data?

•  A key requirement in any unifying Big Data architecture is managing
the complexity of schemas. It seems that we need a new generation
of semantic analysis tools to assist with schema management. What
tools are emerging to support this requirement?


•  Gregory Piatetsky-Shapiro of KDnuggets ran a recent poll on the
largest dataset that his audience of data miners has so far analyzed.
The median size for 2012 was in the range 10-100 GB. If most of the
data for half of the analytics projects can fit into main memory on a
server platform, why is there such a need for expensive
architectures supporting MPP, MapReduce, and the like?

•  http://www.kdnuggets.com/polls/2012/largest-dataset-analyzed-
data-mined.html


!   June: Intelligence
!   July: Disruption
!   August: Analytics
!   September: Integration
!   October: Database
!   November: Cloud


The Comprehensive Approach: A Unified Information Architecture

The Comprehensive Approach: A Unified Information Architecture

Recommended

Recommended

More Related Content

What's hot

What's hot (19)

Similar to The Comprehensive Approach: A Unified Information Architecture

Similar to The Comprehensive Approach: A Unified Information Architecture (20)

More from Inside Analysis

More from Inside Analysis (20)

Recently uploaded

Recently uploaded (20)

The Comprehensive Approach: A Unified Information Architecture