Big Data at Oracle - Strata 2015 San Jose

Take it to the Limit:
An Information Architecture
for Beyond Hadoop
Jeff Pollock
VP, Product Management
February, 2015

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.
#StrataHadoop - Oracle Big Data Architecture

Jeff Pollock speaker bio:
• Product Lead for Big Data
Integration & Governance
• Oracle Middleware
• IBM InfoSphere
• Chief Technical Officer
• Cerebra – Semantic technology startup
• Modulant – Semantic technology startup
• Software Engineer
• Ernst & Young LLP – Java and E-Commerce
• Amgen – Information services
• Author
• Semantic Web for Dummies, 2006
• Adaptive Information, 2004
In this session you’ll learn about how to apply Data Discovery and Deep Data
Storage for new breakthroughs in data warehousing. We’ll discuss the benefits of
using Hadoop technologies like Spark, Kafka, and Hive together with enterprise
information architecture and data governance best practices.
Come hear the latest about Oracle’s Big Data architecture with special emphasis
on new product updates in Oracle Big Data Discovery and Oracle Data Integration
offerings – including the ability to replicate data in real-time with Hadoop as well
as new support for Flume, Kafka, Pig, Oozie, and Spark. You’ll also get the latest
information on Oracle Big Data SQL which can access all sources in NoSQL, HBase,
HDFS, and relational databases in a single query.
The gravity of big data on the information management industry is evolving the
way we collect, store, move, and analyze information. It is both pulling and
pushing the world of business analytics to new breakthroughs in real-time
streaming, predictive capabilities, virtualized data access, and self-service modes
of operation. Join us in this session as we help bridge the gap between old and
new by explaining how to apply data discovery, data governance, and SQL
interoperability in this brave new world of Hadoop technology.
Take it to the Limit: An Information
Architecture for Beyond Hadoop

Customer
Experience
Operational
Improvement
New Business
Models
44% 30% 26%

Traditional
Currency
Photo
Film
Postal
Services
Printing
Press
Record
Industry
Digital
Currency
Web
Publishing
Email Digital
Camera
Digital
Download
Entire Industries are Being Disrupted

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Bigger
Smarter
Easier
CheaperBetter
Faster VALUE
FROM DATA

Connected by Data

Business Data as a Platform

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. | #StrataHadoop - Oracle Big Data Architecture
New Concepts for a Modern Big Data Platform Architecture
Polyglot
Fit for Purpose Data
Lambda
Speed Layer
Batch Layer
Data
Sources
Data
Services
Kappa
Data
Services
Data PipelineData
Sources

Revenge of the Data Nerds!
Lambda
Fit for Purpose Data
Lambda
Speed Layer
Batch Layer
Data
Sources
Data
Services
Lambda
Data
Services
Data PipelineData
Sources

Execution
Innovation
4th Generation Data Architecture for Big Data
WarehouseData FactoryReservoir
Data Streaming
Data Platform
Discovery Lab
Analytics
APIs
Enterprise
Data
Other Data
Sources
Data
Streams
Business
Data
Social/Log
Data
Model First
Analytics
• Reporting-oriented
• Often enterprise wide
in scope, cross LoB
• “you know the
questions to ask”
Reports &
Dashboards
Data First
Analytics
• Data Exploration
• Highly visual and/or
interactive
• “you don’t know the
questions to ask”
Discovery
• Telematics
• Industry Services
• Internet of Things
• Sentiment
Data
Services

Execution
Innovation
Comprehensive Oracle Solution for Big Data
WarehouseFactoryReservoir
Data Streaming
Data Platform
Discovery Lab
Analytics
APIs
Enterprise
Data
Other Data
Sources
Data
Streams
Business
Data
Social/Log
Data
Model First
Analytics
in scope, cross LoB
• “you know the
questions to ask”
Reports &
Dashboards
Data First
Analytics
interactive
questions to ask”
Discovery
• Telematics
• Sentiment
Data
Services
Apache
Oracle
NoSQL
Oracle
CAF & OEP
Oracle Data Integration & Governance
Oracle Database
& Big Data SQL
Oracle
R
Oracle
Big Data
Discovery
Oracle
Business
Intelligence
Oracle
Big Data
Discovery
Apache

Execution
Innovation
Integrated Oracle Engineered Systems for Big Data
Data Streaming
Data Platform
Discovery Lab
Analytics
APIs
Enterprise
Data
Other Data
Sources
Data
Streams
Business
Data
Social/Log
Data
Model First
Analytics
in scope, cross LoB
• “you know the
questions to ask”
Reports &
Dashboards
Data First
Analytics
interactive
questions to ask”
Discovery
• Telematics
• Sentiment
Data
Services
APIs
Analytics

Execution
Innovation
Visionary Oracle Cloud for Big Data
Data Platform
Discovery Lab
Analytics
APIs
Enterprise
Data
Other Data
Sources
Data
Streams
Business
Data
Social/Log
Data
Model First
Analytics
in scope, cross LoB
• “you know the
questions to ask”
Reports &
Dashboards
Data First
Analytics
interactive
questions to ask”
Discovery
• Telematics
• Sentiment
Data
Services
Data Streaming

Oracle Brings Business Value to Big Data
Enterprise-Grade Capabilities
Discover and
Predict – Fast
Govern and Secure
All Data
Simplify Access to
All Data
Performance Integration Availability Scalability Manageability

Enterprise
Productivity
Disruptive
Technology

BIG DATA DISCOVERY

Data Discovery Accelerates Time-to-Value From Data
80% effort typically
spent on evaluating
and preparing data
Data Uncertainty
• Not familiar and overwhelming
• Potential value not obvious
• Requires significant manipulation
Overly dependent on
scarce and highly
skilled resources
Tool Complexity
• Early Hadoop tools only for experts
• Existing BI tools not designed for Hadoop
• Emerging solutions lack broad capabilities

Announcing: Oracle Big Data Discovery
The Visual Face of Hadoop
find explore transform discover share

Find
20
• Access a rich,
interactive catalog of
all data in Hadoop
• Familiar search and
guided navigation for
ease of use
• See data set
summaries, user
annotation and
recommendations
• Provision personal
and enterprise data
to Hadoop via self-
service

Explore
21
• Visualize all
attributes by type
• Sort attributes by
information
potential
• Assess attribute
statistics, data
quality and outliers
• Use scratch pad to
uncover
correlations
between attributes

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. | 2222
• Intuitive, user
driven data
wrangling
• Extensive library of
powerful data
transformations and
enrichments
• Preview results,
undo, commit and
replay transforms
• Test on sample data
then apply to full
data set in Hadoop
Transform

• Join and blend data
for deeper
perspectives
• Compose project
pages via drag and
drop
• Use powerful search
and guided
navigation to ask
questions
• See new patterns in
rich, interactive data
visualizations
Discovery

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. | 24
• Share projects,
bookmarks and
snapshots with
others
• Build galleries and
tell big data stories
• Collaborate and
iterate as a team
• Publish blended
data to HDFS for
leverage in other
tools
Share

Open Platform: Core Technical Innovations on Hadoop
Oracle Big Data Discovery Workloads
Hadoop Cluster
(BDA or Commodity Hardware)
BDD node
data node
data node
data node
data node
name node
Data Processing, Workflow & Monitoring
• Profiling: catalog entry creation, data type &
language detection, schema configuration
• Sampling: dgraph (index) file creation
• Transforms: >100 functions
• Enrichments: location (geo), text (cleanup,
sentiment, entity, key-phrase, whitelist tagging)
Self-Service Provisioning & Data Transfer
• Personal Data: Upload CSV and XLS to HDFS
In-Memory Discovery Indexes
• DGraph: Search, Guided Navigation, Analytics
Studio
• Web UI: Find, Explore, Transform, Discover, Share
Hadoop 2.x
Filesystem
Workload Mgmt
(YARN)
Metadata
(HCatalog)
Other Hadoop Workloads
MR
Spark
Hive
Pig
Oracle Big Data SQL
(Oracle BDA)

Radical Simplicity: Discovery Lab in a Box
Big Data Discovery with the Big Data Appliance X5
• Ready to Go – just add Data
• Pre-configured, Tuned
• Integrated Management
• Run any 3rd party software
• Install Exalytics into a BDA Starter Rack
– In-memory Database & Analytics
– OBIEE Certified with 12c & Big Data SQL
– Big Data Discovery Certified
– Oracle Data Integration Certified

BIG DATA APPLIANCE

18 Sun Oracle X5-2L Servers with per server:
• 2 * 18-core Intel® Xeon® Haswell-EP - fastest Intel processor ever shipped
• 128 GB Memory – DDR4 upgradeable from 128GB (8x16GB) to 768 GB
• 48TB Disk space
Integrated Software (4.1):
• Oracle Linux6.5, Oracle JDK 7u72
• Cloudera Distribution of Apache Hadoop 5.3 – EDH Edition, Cloudera Manager 5.3
• Oracle Big Data SQL 1.1, Oracle R Distribution 3.1.1-2, Oracle NoSQL Database CE 3.2.4
Announcing: BDA X5-2 Update
Faster Processors, More Cores, More Memory, Same Price
BDA X5-2 vs. X4-2 Full Rack
2.25x More Cores 648 cores – Intel® Xeon® E5-2699 v3
2x More Memory Default 3.2TB DDR4-2133MHz
50% More DIMM Slots Up to 13.2TB DDR4-2133MHz

Easier Adoption of New Big Data Technologies
INTEGRATION SKILLS SECURITY
Engineered
Systems
SQL on
All Data
Database
Security on
All Data
SQL

Standard, Modular and Highly Extensible Appliance
 Starter Rack is a fully cabled
and configured for growth with 6
servers
 In-Rack Expansion delivers 6
server modular expansion block
 Full Rack delivers optimal blend
of capacity and expansion
options
 Grow by adding rack – up to 18
racks without additional switches

Driving Business Value from Technology Innovation
Use the Right Tool for the Job and benefit from the Powerof “AND”
RelationalHadoop NoSQL Graph
Run the Business
 Integrate existing
systems
 Mission-critical
tasks
 Use existing
investments
 Ensure skills
relevance
Change the Business
 Disrupt competitors
 Disintermediate
supply chains
 Leverage new
paradigms
 Exploit new analyses
Scale the Business
 Serve data
faster
 Persist data streams
 Meet mobile and
device challenges
 Scale-out
economically
Link the Business
 Associate complex
business entities
 Link to Open Data
 Share data sets via
Ontology
 Evolve data and
schema together

Proven, Cost Effective Solution
“Oracle Big Data Appliance is
an excellent choice for
customers looking to work
with the full suite of
Cloudera’s leading Hadoop-
based technology. It’s more
cost-effective and quicker to
deploy than a DIY cluster.”
⁻ Mike Olson, Cloudera founder, Chief
Strategy Officer, and Chairman of the
Board
Source:
ESG White Paper
21%
Cost Savings
33%
Faster Time
to Value

BIG DATA INTEGRATION

Comprehensive Integration and Governance
Fast
Load
Speed Layer
Batch Layer
Oracle Data Integrator
(Transform)
Oracle GoldenGate
(Move & Ingest)
Data
Governance
Foundation
Enterprise Data Quality
(Profile & Cleanse)
Enterprise Metadata Management & Business Glossary
(Business Glossary, Data Lineage, Impact Analysis and Data Provenance)
Veridata
(Verify)
Data Enrichment
(Prepare)
Real-Time Data Movement
– Low impact capture, stage in Hadoop
– Continuous data availability
Data Transformation
– Bulk data movement
– Pushdown data processing
Data Governance
– Prepare unstructured data
– Profile data with sampling
– Clean data in real time or batch
– Verify data for consistency
– Trace lineage of all data
– Define glossary of business terms

Staging
Open Lambda Architecture with Oracle Big Data Integration
Sqoop
HDFS
Hive
Flume
Capture
Trail
Route
Deliver
Pump
Transformation
Data StreamingKafka (MPP Pub/Sub)
Storm and Trident
Spark Streaming
HBase
Discovery Sandbox/s
ROracle GoldenGate
Oracle Data Governance
Oracle Data Enrichment
Model First
Analytics
in scope, cross LoB
• “you know the
questions to ask”
Data First
Analytics
interactive
questions to ask”
• Telematics
• Sentiment
Reports &
Dashboards
Discovery
Data
Services

Capture
Trail
Route
Deliver
Pump
Announcing: Oracle GoldenGate for Big Data
New DB/
HW/OS/APP
Zero Downtime Upgrades
& Data Migration
Fully Active
Distributed DB
High Availability
& Disaster Recovery
Application
Offloading
Query & Report Offloading
Big Data, DW
& Marts
Real-time BI, Hadoop Data
Staging, Data Ingestion
Event Driven Architecture,
SOA/JMS, Coherence
Message Bus
& Data Grid
Data Synchronization
Across the Enterprise
Global Data
Centers
Real-time Analytics
& Massive Parallelization
Data
Streaming
GoldenGate
Real-time
Data Delivery

Oracle Data Integrator for Transformation on Big Data
Flume
Hive on MR, Tez, Spark
Logs
OLTP DB
SQOOP
OGG
Pig on MR, Tez, Spark
SQOOP
Any DW
OGG
Spark
OEMM
Metadata Mgmt
& Lineage
API/File
Hive/HCat,
HDFS,HBase
Hive/HCat,
HDFS,HBase
NoSQL
Kafka

Business Value of ODI: Low Cost and High Dev Efficiency
Oracle Confidential, under Non-Disclosure 38
No ETL engine is
required
Separation of
Logical and
Physical design
Physical exec on
SQL, Hive, Pig, or
Spark
Runtime exec in
Oozie or via ODI
Java Agent
Rich set of pre-
built operators
User defined
functions

Load to Oracle
OLH/OSCH
Oracle Data Integration on Engineered Systems
Transform
ODI
Hive/HDFS
Federate Hive/HDFS to Oracle
Big Data SQL
Oracle DB
OLTP
Load from Oracle
CopyToBDA
Hive/HDFS
OGGOGG
Hive/HDFS
SQOOP
Flume
Kafka

BIG DATA GOVERNANCE

Data Governance is Not Easy, Hadoop is No Silver Bullet!
Data
Governance
Metadata
Management
Business
Glossary
Data
Profiling
Data
Cleansing
Data
Archiving
Data Privacy
PEOPLE
PROCESS TECHNOLOGY
…people and process first, …tools and capabilities next, …and, there is no magic!
“…the overall impact of poor-
quality data on the whole
dataset remains the same. In
addition, much of the data that
organizations use in a big data
context comes from outside, or
is of unknown structure and
origin. This means that the
likelihood of data quality issues
is even higher than before. So
data quality is actually more
important in the world of big
data."
- Ted Friedman, Gartner
http://www.gartner.com/newsroom/id/2854917

Operational Data Preparation for Data Without Schema
Data Discovery
& Visualization
Enterprise
Reporting
Internet
Logs
Unstructured &
Structured Data
90% of time is
spent WRANGLING
DATA
MONTHS of effort
spent on each new
dataset
PROGRAMERS writing scripts
or complex ETL
Enterprise
ETL & Data
Integration

Internet
Logs
Unstructured &
Structured Data
Data Discovery
& Visualization
Enterprise
Reporting
Enterprise
ETL & Data
Integration
Operational Data Preparation for Data Without Schema

Industry Proven Data Privacy for All Data Types
extend Oracle
Database Security to
Hadoop, NoSQL and
Graph data
authentication and
authorization, audit
and encryption on
Hadoop
end to end data
transparency and
lifecycle management
Oracle Database Security
Industry Standard Access Controls and Lifecycle Controls
Relational
Hadoop
NoSQL
Graph

Comprehensive Metadata Management for Big Data
ETL
BI
Dashboards
App
ETL
ETL
Sys Admin
Executive
BI Developer
Application User
CDC
Data Steward
ETL
Developer
Data Scientist
GG
Oracle Enterprise Metadata Management
Glossary & Catalog Harvest & Stitch Lineage

Most Open & Heterogeneous Big Data Solution
 Hadoop HBase
 Hadoop Hive/Flume
 HP Enscribe
 HP NonStop
 HP Neoview
 Hypersonic SQL
 IBM DB2 i Series
 IBM DB2 UDB
 IBM DB2 z Series
 IBM Informix
 IBM Netezza
 JMS / MQ
 Microsoft Access
 Microsoft SQLServer
 MySQL
 Pivotal Greenplum
 PostgreSQL
 Salesforce.com
 SAP BW / BI
 SAP ERP / ECC
 SAS
 SQL/MP
 SQL/MX
 Sybase ASE
 Sybase IQ
 Teradata
 Adaptive
 Altova
 Apache Hcatalog
 Apache Hive/HQL
 Borland
 CA ERwin
 Cloudera Impala
 COBOL Copybook
 DataStax
 Embarcadero
 EMC ProActivity
 GentleWare
 Google BigQuery
 Grandite
 Hadapt Hive
 Hortonworks Hive
 IBM Cognos
 IBM DB2
 IBM DataStage
 IBM Discovery
 IBM Federation Server
 IBM Lotus Notes
 IBM Netezza
 IBM Rational Rose
 IBM Rational Architect
 Informatica Metadata Mgr.
 Informatica PowerCenter
 CoSORT
 ISO SQL Standard (DDL)
 MapR Hadoop Hive
 MicroFocus
 Microsoft Access
 Microsoft Office Excel
 Microsoft Visio
 Microsoft SQL Server
 Microsoft SSIS
 Microsoft Visual Studio
 Microstrategy
 Magic Draw
 OMG CWM Standard
 OMG UML Standard
 Oracle BI Answers
 Oracle BI Enterprise Edition
 Oracle BI Server
 Oracle DAC
 Oracle Data Integrator
 Oracle Data Modeler
 Oracle Database
 Oracle Designer
 Oracle Hyperion Applications
 Oracle Hyperion Essbase
 Oracle Warehouse Builder
 Pivotal Greenplum
 PostgreSQL
 QlikView
 SAP BO Crystal Reports
 SAP BO Designer
 SAP BO Desktop Intelligence
 SAP BO Repository
 SAP BO Data Integrator
 SAP BO Data Steward
 SAP Master Data Management
 SAP Sybase PowerDesigner
 SAP Sybase ASE Database
 SAS Data Integration Studio
 SAS BI Server
 SAS Information Map
 SAS Metadata Management
 SAS OLAP Server
 Select
 Sparx Architect
 Syncsort
 Tableau
 Talend
 Teradata
 Tigris
 Visible
 W3C DTD & XSD Schema
Operational Integration (Movement / Transformation) Metadata Harvesting (Glossary, Lineage & Impact Analysis)
 Oracle Database
 Oracle Exadata
 Oracle Big Data Appliance
 Oracle TimesTen
 Oracle OLAP
 Oracle Business Intelligence
 Oracle BI Applications
 Oracle E-Business Suite
 Oracle JD Edwards Enterprise One
 Oracle JD Edwards World
 Oracle Fusion Applications
 Oracle Governance Risk and Compliance
 Oracle Fusion AIA
 Oracle Retail Applications
 Oracle Agile BI / DW
 Oracle Agile PLM for Process
 Oracle iFlex FlexCUBE
 Oracle iFlex Mantas
 Oracle Hyperion Applications
 Oracle PeopleSoft
 Oracle Siebel CRM / OnDemand
 Oracle Communications
 Oracle WebLogic Server
 Oracle Coherence Data Grid
 Oracle SOA Suite
 Oracle Enterprise Service Bus
+ open APIs and standards
based meta-model

It’s yours for the taking

Big Data at Oracle - Strata 2015 San Jose

Big Data at Oracle - Strata 2015 San Jose

More Related Content

What's hot

Viewers also liked

Similar to Big Data at Oracle - Strata 2015 San Jose

More from Jeffrey T. Pollock

Recently uploaded

Big Data at Oracle - Strata 2015 San Jose