Hadoop, oracle and the industrial revolution of data

Hadoop, Oracle and theindustrial revolution of dataGuy HarrisonExecutive Director,R&D BusinessIntelligenceSoftware

ValueThe 3-4 V’s Competitive or Community advantage Volume Variety Terabytes Structured Petabytes Unstructured Exabytes Human Generated Zetabytes Machine Generated Velocity User populations x Transaction rates x Machine data © 2012 Quest Software Inc. All rights reserved. Pg. 15

But the vastness is becoming mind bogglingDigital information created 2011 2.13E+21 Total Digital capacity 1.18E+21 Digital information 2008 4.87E+18 Living Human Genomes 5.48E+18 Google 1.10E+17 Human Brain 2.81E+15 1.00E+09 1.00E+10 1.00E+11 1.00E+12 1.00E+13 1.00E+14 1.00E+15 1.00E+16 1.00E+17 1.00E+18 1.00E+19 1.00E+20 1.00E+21 1.00E+22 Gigabyte Terabyte Petabyte Exabyte zettabyte © 2012 Quest Software Inc. All rights reserved. Pg. 17

Samsung Galaxy S IIII specifications Quad-core 1.4 GHz CPU 1GB RAM 64GB Storage 1080p display GSM/Bluetooth/WiFi Network 8MP Camera GPS & Compass © 2012 Quest Software Inc. All rights reserved. Pg. 24

Name: Willy BowmanNationality: GermanDON‟T MENTIONTHE WAR 35

Siri “Siri call me an “I want to jump off a ambulance” bridge” From now on, I‟ll call you „An Ambulance‟. OK? I found 14 bridges nearby:

Big Data AnalyticsHow do we derivevalue from the data? Machine Collective Learning Intelligence Programs that Programs that use evolve with inputs from “crowds‟ “experience” to seem intelligent Predictive Analytics Programs that extrapolate from existing data into the future

Applications Search Optimization Advertising Recommendation • Targeting Systems • Tailoring Security Game optimization Collective • Vulnerability Intelligence • Penetration Detection Medical • Risk analysis Fraud Detection • Diagnosis • Prognosis Predictive Analytics • Churn • Defaults © 2012 Quest Software Inc. All rights reserved. Pg. 61

Map Reduce MAP MAP MAP MAP MAP MAP MAP MAP MAP MAP START MAP REDUCE MAP MAP MAP MAP MAP MAP MAP MAP MAP MAP MAP MAP MAP MAP MAP MAP MAP MAP MAP MAP MAP MAP MAP MAP © 2012 Quest Software Inc. All rights reserved. Pg. 84

Hadoop MAP REDUCE (DISTRIBUTED HADOOP CLIENT (JAVA, PIG, HIVE)Architecture PROCESSING)(1.0) HDFS (DISTRIBUTED STORAGE) JOB TRACKER NAME NODE SECONDARY NAME NODE DATA NODE TASK DATA NODE TASK DATA NODE TASK TRACKER TRACKER TRACKER DATA NODE TASK DATA NODE TASK DATA NODE TASK TRACKER TRACKER TRACKER DATA NODE TASK DATA NODE TASK DATA NODE TASK TRACKER TRACKER TRACKER DATA NODE TASK DATA NODE TASK DATA NODE TASK TRACKER TRACKER TRACKER © 2012 Quest Software Inc. All rights reserved. Pg. 89

Schema on Write Code AnalyseData Extract Transform Load Utilize Cleanse Aggregate Data Warehouse Normalize Schema on Read Code AnalyseData Load Utilize Hadoop Cleanse © 2012 Quest Software Inc. All rights reserved. Pg. 91

Hadoop Oozie (Workflow manager)Ecosystem Hive Pig SQOOP Flume (Query) (Scripting) (RDBMS loader) (Log Loader) ZooKeeper Hbase Hadoop Map Reduce (Locking) (Database) Hadoop File System (HDFS) © 2012 Quest Software Inc. All rights reserved. Pg. 92

HBaseHBase is a real-time database built on Hadoop Log Buffer Cache MemStore Buffer Table Table Table Table Datafiles Redo HFile HFile WA Log ASM HDFS Disks Disks © 2012 Quest Software Inc. All rights reserved. Pg. 94

Hbase Data Model Name Site Counter NameId Name SiteId SiteName Dick Ebay 507,018 1 Dick 1 Ebay Dick Google 690,414 2 Jane 2 Google Jane Google 716,426 3 Facebook Dick Facebook 723,649 4 ILoveLarry.com Jane Facebook 643,261 5 MadBillFans.com Jane ILoveLarry.com 856,767 Dick MadBillFans.com 675,230 NameId SiteId Counter 1 1 507,018 1 3 690,414 2 3 716,426 1 3 723,649 2 3 643,261 2 4 856,767 1 5 675,230 Id Name Ebay Google Facebook (other columns) MadBillFans.com 1 Dick 507,018 690,414 723,649 . . . . . . . . . . . . . . 675,230 Id Name Google Facebook (other columns) ILoveLarry.com 2 Jane 716,426 643,261 . . . . . . . . . . . . . . 856,767

Oracle Big Data Appliance 18 Sun X4270 M2 servers − 48GB RAM per node (864GB total) − 2x6 Core CPU per node (216 total) − 12x2TB HDD per node (216 spindles, 864 TB) − 40Gb/s Infiniband between nodes − 10Gb/s Ethernet to datacentre Competitive Pricing www.oracle.com/us/bigdata/index.html © 2012 Quest Software Inc. All rights reserved. Pg. 108

LatencyOracle’s ORACLE BIG DATA ORACLE EXALOGIC ORACLE EXALYTICSStorage APPLIANCEHierarchy ORACLE WEBLOGIC ORACLE ORACLE NOSQL ESSBASE ORACLE ORACLE EXADATA LOADER FOR HADOOP APACHE ORACLE HADOOP ORACLE RDBMS TIMES TEN Storage Costs © 2012 Quest Software Inc. All rights reserved. Pg. 110

The Big Data Stack DATA SCIENTIST CASCADING R (ET AL) PIG MAHOUT JAVA API JAVA API HIVE MAP-REDUCE HBASE HDFS

The Big Data Stack BIG DATA ANALAYTIC PLATFORM DATA SCIENTIST CASCADING R (ET AL) PIG MAHOUT JAVA API JAVA API HIVE MAP-REDUCE HBASE HDFS

Big Data Analytics Platform INDEXING AND SEARCH SENTIMENT ANALYSIS VISUALIZATION BASKET ANALYSIS RECOMMENDERS BIG DATA ANALYTICS ADVERTISING CLUSTERING OPTIMIZATION CLASSIFICATION EXPERT SYSTEMS (LIKE WATSON)

ETL “Free” Schema on Write Code Analyse Data Extract Transform Load Utilize Cleanse Aggregate Data Warehouse Normalize Schema on Read Code Analyse Data Load Utilize Hadoop Cleanse © 2012 Quest Software Inc. All rights reserved. Pg. 128

SharePlex® for Hadoop Hadoop JMS Queue PosterChange Data Capture Audit / Change Redo-logs Data Batched HDFS HBase Real File Copy Time replication © 2012 Quest Software Inc. All rights reserved. Pg. 140

Summary: The future belongs to those of us prepared to wear funny hats and glasses The connected and mobile internet requires and produces “big data” that is qualitatively different from the data we’ve had before − Requiring different types of datastores Enterprise can leverage big data for competitive advantage − Requiring different types of analytical engines © 2012 Quest Software Inc. All rights reserved. Pg. 144

Hadoop, oracle and the industrial revolution of data

by Guy Harrison on Jan 08, 2013

Accessibility

Upload Details

Usage Rights

Statistics

Hadoop, oracle and the industrial revolution of data Presentation Transcript